Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
2.0.2, 2.1.0
-
None
-
None
Description
If I serialize a type containing a scala Map into a dataframe, it will fail to unserialize it unless I flag the type as a collection.Map in the case class. Else it will make a mismatch between collection.Map (trait) and collection.immutable.Map (default implementation).
case class Person(name: String, details: Map[String, String]) val peopleSeq = Seq( Person("bob", Map("age" -> "30", "address" -> "blabla")), Person("john", Map("age" -> "25", "address" -> "blabla")) ) val peopleDS = peopleSeq.toDS peopleDS.collect should contain theSameElementsAs peopleSeq // failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', // Line 127, Column 40: No applicable constructor/method found for actual parameters "java.lang.String, scala.collection.Map"; // candidates are: "Person(java.lang.String, scala.collection.immutable.Map)"
With this trick it is working :
case class PersonWithForcedMapType(name: String, details: collection.Map[String, String]) val peopleSeq = Seq( PersonWithForcedMapType("bob", Map("age" -> "30", "address" -> "blabla")), PersonWithForcedMapType("john", Map("age" -> "25", "address" -> "blabla")) ) val peopleDS = peopleSeq.toDS peopleDS.collect should contain theSameElementsAs peopleSeq //OK
It is the same if I don't use datasets API but dataframes instead :
val peopleSeq = Seq( Person("bob", Map("age" -> "30", "address" -> "blabla")), Person("john", Map("age" -> "25", "address" -> "blabla")) ) val peopleDF = peopleSeq.toDF val peopleResult = peopleDF.map { row => val name = row.getAs[String](0) val details = row.getAs[Map[String, String]](1) Person(name, details) } peopleResult.collect should contain theSameElementsAs peopleSeq // failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', // Line 127, Column 40: No applicable constructor/method found for actual parameters "java.lang.String, scala.collection.Map"; // candidates are: "Person(java.lang.String, scala.collection.immutable.Map)"
Attachments
Issue Links
- duplicates
-
SPARK-18891 Support for specific collection types
- Resolved