Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39496

Inline eval path cannot handle null structs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.3, 3.2.1, 3.3.0, 3.4.0
    • 3.1.3, 3.2.2, 3.4.0, 3.3.1
    • SQL
    • None

    Description

      This issue is somewhat similar to SPARK-39061, but for the eval path rather than the codegen path.

      Example:

      set spark.sql.codegen.wholeStage=false;
      select inline(array(named_struct('a', 1, 'b', 2), null));
      

      This results in a NullPointerException:

      22/06/16 15:10:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
      java.lang.NullPointerException
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
      	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
      

      The next example doesn't require setting spark.sql.codegen.wholeStage to false:

      val dfWide = (Seq((1))
        .toDF("col0")
        .selectExpr(Seq.tabulate(99)(x => s"$x as col${x + 1}"): _*))
      
      val df = (dfWide
        .selectExpr("*", "array(named_struct('a', 1, 'b', 2), null) as struct_array"))
      
      df.selectExpr("*", "inline(struct_array)").collect
      

      The result is similar:

      22/06/16 15:18:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
      java.lang.NullPointerException
      	at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:80)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_8$(Unknown Source)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
      	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
      

      Attachments

        Activity

          People

            bersprockets Bruce Robbins
            bersprockets Bruce Robbins
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: