Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19104

CompileException with Map and Case Class in Spark 2.1.0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0, 2.2.0
    • 2.1.2, 2.2.0
    • Optimizer, SQL
    • None

    Description

      The following code will run with Spark 2.0.2 but not with Spark 2.1.0:

      case class InnerData(name: String, value: Int)
      case class Data(id: Int, param: Map[String, InnerData])
      
      val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + 100))))
      val ds   = spark.createDataset(data)
      

      Exception:

      Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 63, Column 46: Expression "ExternalMapToCatalyst_value_isNull1" is not an rvalue 
        at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) 
        at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) 
        at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) 
        at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) 
        at org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) 
        at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) 
        at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) 
        at org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) 
        at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) 
        at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) 
        at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) 
        at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) 
        at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) 
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) 
        at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) 
        at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) 
        at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) 
        at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) 
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) 
        at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) 
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) 
        at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) 
        at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) 
        at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) 
        at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) 
        at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) 
        at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) 
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) 
        at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) 
        at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) 
        at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) 
        at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) 
        at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) 
        at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) 
        at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) 
        at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) 
        at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) 
        at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) 
        at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) 
        at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196) 
        at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91) 
        at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:935) 
        ... 77 more 
      

      Attachments

        Issue Links

          Activity

            People

              viirya L. C. Hsieh
              nils.grabbert Nils Grabbert
              Votes:
              6 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: