Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235 Push-based Shuffle Improvement Tasks
  3. SPARK-38093

Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.1
    • None
    • Shuffle
    • None

    Description

      Currently we are setting shuffleMergeAllowed to false before prepareShuffleServicesForShuffleMapStage if the shuffle dependency is already finalized. Ideally it is better to do it right after shuffle dependency finalization for a determinate stage. cc mridulm80

      Attachments

        Activity

          People

            Unassigned Unassigned
            vsowrirajan Venkata krishnan Sowrirajan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: