[SPARK-19089] Support nested arrays/seqs in Datasets

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Michal Šenkýř
• Reporter: Michal Šenkýř
• Created: 05/Jan/17 18:49
• Updated: 22/May/17 08:51
• Resolved: 22/May/17 08:51

Description

Nested arrays and seqs are not supported in Datasets:

scala> spark.createDataset(Seq(Array(Array(1))))
<console>:24: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
       spark.createDataset(Seq(Array(Array(1))))
                          ^

scala> Seq(Array(Array(1))).toDS()
<console>:24: error: value toDS is not a member of Seq[Array[Array[Int]]]
       Seq(Array(Array(1))).toDS()

scala> spark.createDataset(Seq(Seq(Seq(1))))
<console>:24: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
       spark.createDataset(Seq(Seq(Seq(1))))

scala> Seq(Seq(Seq(1))).toDS()
<console>:24: error: value toDS is not a member of Seq[Seq[Seq[Int]]]
       Seq(Seq(Seq(1))).toDS()

Issue Links

• is related to: SPARK-18891 Support for specific collection types
• links to: [Github] Pull Request #18011 (michalsenkyr)