Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28732

org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java' when storing the result of a count aggregation in an integer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0, 2.2.0, 2.3.0, 2.4.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      I am using agg function on a dataset, and i want to count the number of lines upon grouping columns. I would like to store the result of this count in an integer, but it fails with this output : 

      [ERROR]: org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator - failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 89, Column 53: No applicable constructor/method found for actual parameters "long"; candidates are: "java.lang.Integer(int)", "java.lang.Integer(java.lang.String)"
      
      Here is the line 89 and a few others to understand :
      
      /* 085 */ long value13 = i.getLong(5);
       /* 086 */ argValue4 = value13;
       /* 087 */
       /* 088 */
       /* 089 */ final java.lang.Integer value12 = false ? null : new java.lang.Integer(argValue4);
      

       

      As per Integer documentation, there is not constructor for the type Long, so this is why the generated code fails.

      Here is my code : 

      org.apache.spark.sql.Dataset<row2Struct> ds_row2 = ds_conntAggregateRow_1_Out_1
       .groupBy(org.apache.spark.sql.functions.col("n_name").as("n_nameN"),
       org.apache.spark.sql.functions.col("o_year").as("o_yearN"))
       .agg(org.apache.spark.sql.functions.count("n_name").as("countN"),
       .as(org.apache.spark.sql.Encoders.bean(row2Struct.class));
      

      row2Struct class is composed of n_nameN: String, o_yearN: String, countN: Int

      If countN is a Long, code above wont fail

      If it is an Int, it works in 1.6 and 2.0, but fails on version 2.1+

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ametivier Alix Métivier
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: