Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22222

Fix the ARRAY_MAX in BufferHolder and add a test

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • Spark Core
    • None

    Description

      This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to `Int.MaxValue - 8`. It is not a valid number because it will cause the following line fail when such a large byte array is allocated: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 We need to make sure the new length is a multiple of 8.

      We need to add one test for the fix. Note that the test should work independently with the heap size of the test JVM.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            fengliu@databricks.com Feng Liu Assign to me
            liufeng.ee@gmail.com Feng Liu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment