Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27186

Optimize SortShuffleWriter writing process

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 3.0.0
    • None
    • Shuffle, Spark Core
    • None

    Description

      If the SortShuffleWriter.write method records is empty, it should be return directly and no need to keep running,refer to the process of BypassMergeSortShuffleWriter.write. Here are the benefits of create Instance ExternalSorter and tmp file. 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              wangjch wangjiaochun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: