Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7271 Redesign shuffle interface for binary processing
  3. SPARK-10708

Consolidate SortShuffleManager and UnsafeShuffleManager

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • Shuffle, Spark Core
    • None

    Description

      There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-10710 Remove ability to set spark.shuffle.spill=false and spark.sql.planner.externalSort=false

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SPARK-10403 UnsafeRowSerializer can't work with UnsafeShuffleManager (tungsten-sort)

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #8829 [Github] Pull Request #8829 (JoshRosen)

          Activity

            People

              joshrosen Josh Rosen
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: