Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40403

Negative size in error message when unsafe array is too big



    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None


      When initializing an overly large unsafe array via UnsafeArrayWriter#initialize, BufferHolder#grow may report an error message with a negative size, e.g.:

      java.lang.IllegalArgumentException: Cannot grow BufferHolder by size -2115263656 because the size is negative

      (Note: This is not related to SPARK-39608, as far as I can tell, despite having the same symptom).

      When calculating the initial size in bytes needed for the array, UnsafeArrayWriter#initialize uses an int expression, which can overflow. The initialize method then passes the negative size to BufferHolder#grow, which complains about the negative size.

      Example (the following will run just fine on a 16GB laptop, despite the large driver size setting):

      bin/spark-sql --driver-memory 22g --master "local[1]"
      create or replace temp view data1 as
      select 0 as key, id as val
      from range(0, 268271216);
      create or replace temp view data2 as
      select key as lkey, collect_list(val) as bigarray
      from data1
      group by key;
      -- the below cache forces Spark to create unsafe rows
      cache lazy table data2;
      select count(*) from data2;

      After a few minutes, BufferHolder#grow will throw the following exception:

      java.lang.IllegalArgumentException: Cannot grow BufferHolder by size -2115263656 because the size is negative
      	at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:67)
      	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter.initialize(UnsafeArrayWriter.java:61)
      	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
      	at org.apache.spark.sql.catalyst.expressions.aggregate.Collect.serialize(collect.scala:73)
      	at org.apache.spark.sql.catalyst.expressions.aggregate.Collect.serialize(collect.scala:37)

      This query was going to fail anyway, but the message makes it looks like a bug in Spark rather than a user problem. UnsafeArrayWriter#initialize should calculate using a long expression and fail if the size exceeds Integer.MAX_VALUE, showing the actual initial size in the error message.




            bersprockets Bruce Robbins
            bersprockets Bruce Robbins
            0 Vote for this issue
            2 Start watching this issue