[SPARK-30158] Resolve Array + reference type compile problems in 2.13, with sc.parallelize

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, SQL
• Assignee: Sean R. Owen
• Reporter: Sean R. Owen
• Created: 06/Dec/19 20:49
• Updated: 09/Dec/19 20:42
• Resolved: 09/Dec/19 20:42

Description

Scala 2.13 has some different rules about resolving Arrays as Seqs when the array is of a reference type. This primarily affects calls to sc.parallelize(Array(...)) where elements aren't primitive:

[ERROR] [Error] /Users/seanowen/Documents/spark_2.13/mllib/src/main/scala/org/apache/spark/mllib/pmml/PMMLExportable.scala:61: overloaded method value apply with alternatives:
  (x: Unit,xs: Unit*)Array[Unit] <and>
  (x: Double,xs: Double*)Array[Double] <and>
  ...

This is easy to resolve by using Seq instead.

Closely related: WrappedArray is a type def in 2.13, which makes it unusable in Java. One set of tests needs to adapt.

Issue Links

• links to: GitHub Pull Request #26787