All Questions
8,812
questions
0
votes
1
answer
17
views
Adding new Rows to Spark Partition while using forEachPartition
I am trying to add a new Row to each Partition in my Spark Job. I am using the following code to achieve this:
StructType rowType = new StructType();
rowType.add(DataTypes.createStructField("...
- 1,081
0
votes
0
answers
33
views
IntelliJ Scala 2.13.0 no longer compiles - Error compiling the sbt component 'compiler-bridge-2.13.0-66.0'
I am working on a hobby project on my desktop using Scala 2.13.0 and JDK 22. The project compiles and runs perfectly fine there. This project is also on my personal GitHub. Because I went on holiday ...
-1
votes
0
answers
39
views
Using spark 3.4.1 lib in Java when extending StringRegexExpression to a java class
I am using spark 3.4.1 in maven project where I am configured scala (2.13.8) lang as well. I am trying to create a class Like.java in project by extending spark's StringRegexExpression
package com....
- 66
-1
votes
0
answers
57
views
Scala/SBT/Java - When running app locally with java 21 getting InaccessibleObjectException
Hi all I am using scala version 2.12.18, sbt version 1.9.0 and java jdk 21.0.3. When trying to run my application locally through
sbt run -Dconfig.resource=local-qa.conf
I am getting this error:
[...
- 51
0
votes
0
answers
76
views
Importing libraries with same names
I have two libraries that both follow the same names for importing, but one has more functionality than the other. How do I make scala choose the right library to use when importing? Currently, it’s ...
0
votes
0
answers
150
views
Is it safe run blocking call in OS executor when using virtual thread?
When using virtual thread, can I safely use a separate thread pool while interacting with a library we don't have a control over, such as HikariCP DB connection manager, which uses synchronized blocks,...
- 2,541
0
votes
0
answers
37
views
For any given project built using SBT, is there any way, by creating a plugin/other ways, to get and STORE the details of modules and dependencies
I have a sample SBT built JAVA project Apache Pekko v1.0.0
https://github.com/apache/pekko/tree/v1.0.0
I need to setup automation in such a way that I get the name of the modules in this project, or ...
-1
votes
1
answer
52
views
issue with unit-testing spark scala project related to json4s
I'm using scala 2.12, spark 3.1.2, and jdk 1.8, and I'm facing what looks to be a dependency issue: A needed class was not found. This could be due to an error in your runpath. Missing class: org/...
- 2,926
0
votes
1
answer
68
views
DataFrameWriter.partitionBy does not guarantee sort order
I have a dataset that have the following schema:
(timestamp, partitionKey, logValue)
I want to have the dataset to be sorted by timestamp, but write to file in the follow directory layout:
outputDir/...
- 1
1
vote
0
answers
41
views
use java udf in pyspark, raise type exception
Step 1: Define a scala UDF:
import org.apache.spark.sql.api.java.UDF1
import scala.collection.mutable
class GetMidVal extends UDF1[mutable.WrappedArray[Double], Double] {
override def call(arr: ...
- 11
0
votes
1
answer
134
views
java.lang.OutOfMemoryError: UTF16 String size exceeding default value
I was trying to load a tsv files from urls (max file size was 1.05 GB or 1129672402 Bytes)
I used java.net.URL for it.
But, it throwed the below error (for the largest one)-
java.lang.OutOfMemoryError:...
- 13
0
votes
0
answers
27
views
Scalafx: UnsupportedClassVersionError [duplicate]
I'm trying to use Scalafx in project.
Dependencies are:
ThisBuild / scalaVersion := "2.12.1"
libraryDependencies += "com.google.code.findbugs" % "jsr305" % "3....
- 1,178
0
votes
0
answers
46
views
Serialization.writePretty format JSON the same way that pre-commit-hooks pretty-format-json does
I'm coding in Scala, using these packages package org.json4s package jackson and the function writePretty of Serialization class.
The thing is it's not formatted the way that pre-commit-hooks pretty-...
1
vote
1
answer
77
views
Are there Scala 3 Specific enum Equivalents of Java's EnumSet/EnumMap?
Specifically using the new enum keyword provided by Scala 3...
enum Translation(val bit: Byte):
case FlipX extends Translation(1)
case FlipY extends Translation(2)
case RotateClockwise ...
- 5,793
-1
votes
1
answer
39
views
How can I set environment variables via gradle script
I have a gradle script that in the test section sets a lot of testing environment variables. These testing variables are used across different packages and I would like to extract this into a separate ...
- 1,069