Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22222

Fix the ARRAY_MAX in BufferHolder and add a test

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to `Int.MaxValue - 8`. It is not a valid number because it will cause the following line fail when such a large byte array is allocated: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 We need to make sure the new length is a multiple of 8.

      We need to add one test for the fix. Note that the test should work independently with the heap size of the test JVM.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fengliu@databricks.com Feng Liu
                Reporter:
                liufeng.ee@gmail.com Feng Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: