Questions tagged [scala]
Scala is a general-purpose programming language principally targeting the Java Virtual Machine. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles. Its key features are: an advanced static type system with type inference; function types; pattern-matching; implicit parameters and conversions; operator overloading; full interoperability with Java; concurrency
scala
13,309
questions
197
votes
10
answers
181k
views
How to select the first row of each group?
I have a DataFrame generated as follow:
df.groupBy($"Hour", $"Category")
.agg(sum($"value") as "TotalValue")
.sort($"Hour".asc, $"TotalValue".desc))
The results look like:
+----+--------+-------...
- 8,244
611
votes
7
answers
172k
views
What are all the uses of an underscore in Scala?
I've taken a look at the list of surveys taken on scala-lang.org and noticed a curious question: "Can you name all the uses of “_”?". Can you? If yes, please do so here. Explanatory examples are ...
- 63.8k
281
votes
12
answers
84k
views
Difference between method and function in Scala
I read Scala Functions (part of Another tour of Scala). In that post he stated:
Methods and functions are not the same thing
But he didn't explain anything about it. What was he trying to say?
- 10.3k
438
votes
2
answers
58k
views
Where does Scala look for implicits?
An implicit question to newcomers to Scala seems to be: where does the compiler look for implicits? I mean implicit because the question never seems to get fully formed, as if there weren't words for ...
- 296k
390
votes
11
answers
85k
views
How do I get around type erasure on Scala? Or, why can't I get the type parameter of my collections?
It's a sad fact of life on Scala that if you instantiate a List[Int], you can verify that your instance is a List, and you can verify that any individual element of it is an Int, but not that it is a ...
- 296k
41
votes
7
answers
21k
views
Resolving dependency problems in Apache Spark
The common problems when building and deploying Spark applications are:
java.lang.ClassNotFoundException.
object x is not a member of package y compilation errors.
java.lang.NoSuchMethodError
How ...
- 1,702
79
votes
3
answers
86k
views
Querying Spark SQL DataFrame with complex types
How Can I query an RDD with complex types such as maps/arrays?
for example, when I was writing this test code:
case class Test(name: String, map: Map[String, String])
val map = Map("hello" -> "...
- 2,566
167
votes
9
answers
101k
views
How to store custom objects in Dataset?
According to Introducing Spark Datasets:
As we look forward to Spark 2.0, we plan some exciting improvements to Datasets, specifically:
...
Custom encoders – while we currently autogenerate ...
192
votes
16
answers
59k
views
How to define "type disjunction" (union types)?
One way that has been suggested to deal with double definitions of overloaded methods is to replace overloading with pattern matching:
object Bar {
def foo(xs: Any*) = xs foreach {
case _:...
- 21k
232
votes
4
answers
39k
views
Scala 2.8 breakOut
In Scala 2.8, there is an object in scala.collection.package.scala:
def breakOut[From, T, To](implicit b : CanBuildFrom[Nothing, T, To]) =
new CanBuildFrom[From, T, To] {
def apply(from: ...
- 134k
289
votes
1
answer
43k
views
What are Scala context and view bounds?
In a simple way, what are context and view bounds and what is the difference between them?
Some easy-to-follow examples would be great too!
- 3,400
243
votes
4
answers
65k
views
What does `:_*` (colon underscore star) do in Scala?
I have the following piece of code from this question:
def addChild(n: Node, newChild: Node) = n match {
case Elem(prefix, label, attribs, scope, child @ _*) => Elem(prefix, label, attribs, ...
- 15.6k
258
votes
11
answers
254k
views
Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
Getting strange behavior when calling function outside of a closure:
when function is in a object everything is working
when function is in a class get :
Task not serializable: java.io....
- 9,895
173
votes
16
answers
437k
views
Write single CSV file using spark-csv
I am using https://github.com/databricks/spark-csv , I am trying to write a single CSV, but not able to, it is making a folder.
Need a Scala function which will take parameter like path and file name ...
- 3,285
69
votes
10
answers
86k
views
Write to multiple outputs by key Spark - one Spark job
How can you write to multiple outputs dependent on the key using Spark in a single Job.
Related: Write to multiple outputs by key Scalding Hadoop, one MapReduce Job
E.g.
sc.makeRDD(Seq((1, "a"), (1,...
- 31.3k