Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7542

Support off-heap sort buffer in UnsafeExternalSorter

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.6.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      UnsafeExternalSorter, introduced in SPARK-7081, uses on-heap long[] arrays as its sort buffers. When records are small, the sorting array might be as large as the data pages, so it would be useful to be able to allocate this array off-heap (using our unsafe LongArray). Unfortunately, we can't currently do this because TimSort calls allocate() to create data buffers but doesn't call any corresponding cleanup methods to free them.

      We should look into extending TimSort with buffer freeing methods, then consider switching to LongArray in UnsafeShuffleSortDataFormat.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'viirya' has created a pull request for this issue:
          https://github.com/apache/spark/pull/8068

          Show
          apachespark Apache Spark added a comment - User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/8068
          Hide
          apachespark Apache Spark added a comment -

          User 'davies' has created a pull request for this issue:
          https://github.com/apache/spark/pull/9477

          Show
          apachespark Apache Spark added a comment - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/9477
          Hide
          joshrosen Josh Rosen added a comment -

          Issue resolved by pull request 9477
          https://github.com/apache/spark/pull/9477

          Show
          joshrosen Josh Rosen added a comment - Issue resolved by pull request 9477 https://github.com/apache/spark/pull/9477

            People

            • Assignee:
              davies Davies Liu
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development