Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-34826

Adaptive fetch of shuffle mergers for Push based shuffle

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      Currently the shuffle mergers are set during the creation of ShuffleMapStage. In the initial set of stages, there won't be enough executors added which can cause not enough shuffle mergers to be set during the creation of the shuffle map stage. This task is to handle the issue of low merge ratio for initial stages.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vsowrirajan Venkata krishnan Sowrirajan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: