[SPARK-16815] Dataset[List[T]] leads to ArrayStoreException

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Michal Šenkýř
• Reporter: TobiasP
• Created: 30/Jul/16 08:54
• Updated: 06/Jan/17 08:10
• Resolved: 06/Jan/17 08:10

Description

scala> spark.sqlContext.createDataset(sc.parallelize(List(1) :: Nil)).collect
java.lang.ArrayStoreException: scala.collection.mutable.WrappedArray$ofRef      
  at scala.collection.mutable.ArrayBuilder$ofRef.$plus$eq(ArrayBuilder.scala:87)
  at scala.collection.mutable.ArrayBuilder$ofRef.$plus$eq(ArrayBuilder.scala:56)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
  at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2218)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
  at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2568)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2217)
  at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:2222)
  at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:2222)
  at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2581)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2222)
  at org.apache.spark.sql.Dataset.collect(Dataset.scala:2198)
  ... 48 elided

Issue Links

• is related to: SPARK-16792 Dataset containing a Case Class with a List type causes a CompileException (converting sequence to list)