Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41535

InterpretedUnsafeProjection and InterpretedMutableProjection can corrupt unsafe buffer when used with calendar interval data

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.1, 3.2.3, 3.4.0
    • 3.2.3, 3.3.2, 3.4.0
    • SQL
    • None

    Description

      This returns the wrong answer:

      set spark.sql.codegen.wholeStage=false;
      set spark.sql.codegen.factoryMode=NO_CODEGEN;
      
      select first(col1), last(col2) from values
      (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0))
      as data(col1, col2);
      
      +---------------+---------------+
      |first(col1)    |last(col2)     |
      +---------------+---------------+
      |16 years 2 days|16 years 2 days|
      +---------------+---------------+
      

      In the above case, TungstenAggregationIterator uses InterpretedUnsafeProjection to create the aggregation buffer and then initializes all the fields to null. InterpretedUnsafeProjection incorrectly calls UnsafeRowWriter#setNullAt, rather than unsafeRowWriter#write, for the two calendar interval fields. As a result, the writer never allocates memory from the variable length region for the two decimals, and the pointers in the fixed region get left as zero. Later, when InterpretedMutableProjection attempts to update the first field, UnsafeRow#setInterval picks up the zero pointer and stores interval data on top of the null-tracking bit set. The call to UnsafeRow#setInterval for the second field also stomps the null-tracking bit set. Later updates to the null-tracking bit set (e.g., calls to setNotNullAt) further corrupt the interval data, turning interval 7 years 2 days into interval 16 years 2 days.

      Even if you fix the above bug to InterpretedUnsafeProjection so that the buffer is created correctly, InterpretedMutableProjection has a similar bug to SPARK-41395, except this time for calendar interval data:

      set spark.sql.codegen.wholeStage=false;
      set spark.sql.codegen.factoryMode=NO_CODEGEN;
      
      select first(col1), last(col2), max(col3) from values
      (null, null, 1),
      (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
      as data(col1, col2, col3);
      
      +---------------+---------------+---------+
      |first(col1)    |last(col2)     |max(col3)|
      +---------------+---------------+---------+
      |16 years 2 days|16 years 2 days|3        |
      +---------------+---------------+---------+
      

      These two bugs could get exercised during codegen fallback. Take for example this case where I forced codegen to fail for the Greatest expression:

      spark-sql> select first(col1), last(col2), max(col3) from values
      (null, null, 1),
      (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
      as data(col1, col2, col3);
      
      22/12/15 13:06:23 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 70, Column 1: ';' expected instead of 'if'
      ...
      22/12/15 13:06:24 WARN MutableProjection: Expr codegen error and falling back to interpreter mode
      java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 78, Column 1: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 78, Column 1: ';' expected instead of 'boolean'
      ...
      16 years 2 days	16 years 2 days	3
      Time taken: 5.852 seconds, Fetched 1 row(s)
      spark-sql> 
      

      Attachments

        Activity

          People

            bersprockets Bruce Robbins
            bersprockets Bruce Robbins
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: