Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27186

Optimize SortShuffleWriter writing process

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Shuffle
    • Labels:
      None

      Description

      If the SortShuffleWriter.write method records is empty, it should be return directly and no need to keep running,refer to the process of BypassMergeSortShuffleWriter.write. Here are the benefits of create Instance ExternalSorter and tmp file. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                wangjch wangjiaochun
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: