[SPARK-18055] Dataset.flatMap can't work with types from customized jar

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.0.3, 2.1.1, 2.2.0
• Component/s: SQL
• Target Version/s: 2.2.0
• Assignee: Michael Armbrust
• Reporter: Davies Liu
• Created: 21/Oct/16 17:46
• Updated: 12/Apr/17 13:19
• Resolved: 08/Mar/17 09:35

Description

Try to apply flatMap() on Dataset column which of of type
com.A.B

Here's a schema of a dataset:

root
 |-- id: string (nullable = true)
 |-- outputs: array (nullable = true)
 |    |-- element: string

flatMap works on RDD

 ds.rdd.flatMap(_.outputs)

flatMap doesnt work on dataset and gives the following error

ds.flatMap(_.outputs)

The exception:

scala.ScalaReflectionException: class com.A.B in JavaMirror … not found
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
    at line189424fbb8cd47b3b62dc41e417841c159.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$typecreator3$1.apply(<console>:51)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
    at org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125)

Spoke to Michael Armbrust and he confirmed it as a Dataset bug.

There is a workaround using explode()

ds.select(explode(col("outputs")))

Issue Links

• is duplicated by: SPARK-18139 Dataset mapGroups with return typ Seq[Product] produces scala.ScalaReflectionException: object $line262.$read not found
• is duplicated by: SPARK-17890 scala.ScalaReflectionException
• links to: [Github] Pull Request #17201 (marmbrus)