[SPARK-22222] Fix the ARRAY_MAX in BufferHolder and add a test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Feng Liu
• Reporter: Feng Liu
• Created: 09/Oct/17 05:28
• Updated: 09/Apr/18 20:49
• Resolved: 10/Oct/17 04:42

Description

This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to `Int.MaxValue - 8`. It is not a valid number because it will cause the following line fail when such a large byte array is allocated: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 We need to make sure the new length is a multiple of 8.

We need to add one test for the fix. Note that the test should work independently with the heap size of the test JVM.

Issue Links

• is related to: SPARK-22033 BufferHolder, other size checks should account for the specific VM array size limitations
• links to: [Github] Pull Request #19460 (liufengdb)
• links to: [Github] Pull Request #19697 (dongjoon-hyun)