Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18800

Correct the assert in UnsafeKVExternalSorter which ensures array size

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.2.0
    • SQL
    • None

    Description

      UnsafeKVExternalSorter uses UnsafeInMemorySorter to sort the records of BytesToBytesMap if it is given a map.

      Currently we use the number of keys in BytesToBytesMap to determine if the array used for sort is enough or not. We has an assert that ensures the size of the array is enough: map.numKeys() <= map.getArray().size() / 2.

      However, each record in the map takes two entries in the array, one is record pointer, another is key prefix. So the correct assert should be map.numKeys() * 2 <= map.getArray().size() / 2.

      Attachments

        Activity

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: