[SPARK-17368] Scala value classes create encoder problems and break at runtime

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.2, 2.0.0
• Fix Version/s: 2.1.0
• Component/s: Spark Core, SQL

Environment

JDK 8 on MacOS
Scala 2.11.8
Spark 2.0.0

• Assignee: Jakob Odersky
• Reporter: Aris Vlasakakis
• Created: 01/Sep/16 23:12
• Updated: 25/Aug/18 09:09
• Resolved: 14/Oct/16 00:48

Description

Using Scala value classes as the inner type for Datasets breaks in Spark 2.0 and 1.6.X.

This simple Spark 2 application demonstrates that the code will compile, but will break at runtime with the error. The value class is of course FeatureId, as it extends AnyVal.

Exception in thread "main" java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: Couldn't find v on int
assertnotnull(input[0, int, true], top level non-flat input object).v AS v#0
+- assertnotnull(input[0, int, true], top level non-flat input object).v
   +- assertnotnull(input[0, int, true], top level non-flat input object)
      +- input[0, int, true]".
        at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:279)
        at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421)
        at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421)

Test code for Spark 2.0.0:

import org.apache.spark.sql.{Dataset, SparkSession}

object BreakSpark {
  case class FeatureId(v: Int) extends AnyVal

  def main(args: Array[String]): Unit = {
    val seq = Seq(FeatureId(1), FeatureId(2), FeatureId(3))
    val spark = SparkSession.builder.getOrCreate()
    import spark.implicits._
    spark.sparkContext.setLogLevel("warn")
    val ds: Dataset[FeatureId] = spark.createDataset(seq)
    println(s"BREAK HERE: ${ds.count}")
  }
}

Issue Links

• is duplicated by: SPARK-17367 Cannot define value classes in REPL
• relates to: SPARK-20384 supporting value classes over primitives in DataSets
• links to: [Github] Pull Request #15284 (jodersky)