[SPARK-25776] The disk write buffer size must be greater than 12.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: liuxian
• Reporter: liuxian
• Created: 19/Oct/18 02:12
• Updated: 26/Apr/19 22:25
• Resolved: 04/Nov/18 16:58

Description

In UnsafeSorterSpillWriter.java, when we write a record to a spill file wtih {{ void write(Object baseObject, long baseOffset, int recordLength, long keyPrefix)}}, recordLength and keyPrefix will be written the disk write buffer first, and these will take 12 bytes, so the disk write buffer size must be greater than 12.

If diskWriteBufferSize is 10, it will print this exception info:

java.lang.ArrayIndexOutOfBoundsException: 10
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.writeLongToBuffer (UnsafeSorterSpillWriter.java:91)
at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.write(UnsafeSorterSpillWriter.java:123)
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spillIterator(UnsafeExternalSorter.java:498)
at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:222)
at org.apache.spark.memory.MemoryConsumer.spill(MemoryConsumer.java:65)

Issue Links

• links to: [Github] Pull Request #22754 (10110346)
• links to: [Github] Pull Request #22754 (10110346)
• links to: GitHub Pull Request #22754