Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25776

The disk write buffer size must be greater than 12.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      In UnsafeSorterSpillWriter.java, when we write a record to a spill file wtih {{ void write(Object baseObject, long baseOffset, int recordLength, long keyPrefix)}}, recordLength and keyPrefix will be written the disk write buffer first, and these will take 12 bytes, so the disk write buffer size must be greater than 12.

      If diskWriteBufferSize is 10, it will print this exception info:

      java.lang.ArrayIndexOutOfBoundsException: 10
      at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.writeLongToBuffer (UnsafeSorterSpillWriter.java:91)
      at org.apache.spark.util.collection.unsafe.sort.UnsafeSorterSpillWriter.write(UnsafeSorterSpillWriter.java:123)
      at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spillIterator(UnsafeExternalSorter.java:498)
      at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:222)
      at org.apache.spark.memory.MemoryConsumer.spill(MemoryConsumer.java:65)

        Attachments

          Activity

            People

            • Assignee:
              10110346 liuxian
              Reporter:
              10110346 liuxian
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: