[SPARK-19104] CompileException with Map and Case Class in Spark 2.1.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 2.1.2, 2.2.0
• Component/s: Optimizer, SQL
• Target Version/s: 2.3.0
• Assignee: L. C. Hsieh
• Reporter: Nils Grabbert
• Created: 06/Jan/17 10:18
• Updated: 18/Jul/17 01:39
• Resolved: 27/Jun/17 17:00

Description

The following code will run with Spark 2.0.2 but not with Spark 2.1.0:

case class InnerData(name: String, value: Int)
case class Data(id: Int, param: Map[String, InnerData])

val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + 100))))
val ds   = spark.createDataset(data)

Exception:

Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 63, Column 46: Expression "ExternalMapToCatalyst_value_isNull1" is not an rvalue 
  at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) 
  at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) 
  at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) 
  at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) 
  at org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) 
  at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) 
  at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) 
  at org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) 
  at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) 
  at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) 
  at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) 
  at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) 
  at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) 
  at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) 
  at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) 
  at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) 
  at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) 
  at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) 
  at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) 
  at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) 
  at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) 
  at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) 
  at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) 
  at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) 
  at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) 
  at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) 
  at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) 
  at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) 
  at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) 
  at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) 
  at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) 
  at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) 
  at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) 
  at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) 
  at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) 
  at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) 
  at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) 
  at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) 
  at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) 
  at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196) 
  at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91) 
  at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:935) 
  ... 77 more 

Issue Links

• duplicates: SPARK-18891 Support for specific collection types
• is duplicated by: SPARK-21391 Cannot convert a Seq of Map whose value type is again a seq, into a dataset
• links to: [Github] Pull Request #18418 (viirya)
• links to: [Github] Pull Request #18626 (kiszk)
• links to: [Github] Pull Request #18627 (kiszk)