Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33235

Push-based Shuffle Improvement Tasks

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Shuffle, Spark Core

    Description

      This is the parent jira for follow-up improvement tasks for supporting Push-based shuffle. Refer SPARK-30602.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-30602 SPIP: Support push-based shuffle to improve shuffle efficiency

          • Major - Major loss of function.
          • Resolved
          1.
          		 Enable Push-based shuffle service to store state in NM level DB for work preserving restart Sub-task Resolved Ye Zhou
          2.
          		 Limit the number of pending blocks in memory and store blocks that collide Sub-task Open Unassigned
          3.
          		 Pluggable API to fetch shuffle merger locations with Push based shuffle Sub-task Open Unassigned
          4.
          		 Better heuristics to compute number of shuffle mergers required for a ShuffleMapStage Sub-task Open Unassigned
          5.
          		 Server side metrics related to push-based shuffle Sub-task Resolved Minchu Yang
          6.
          		 Improve locality for push-based shuffle especially for join like operations Sub-task In Progress Unassigned
          7.
          		 Enable ShuffleBlockPusher to stop pushing blocks for a particular shuffle partition Sub-task Open Unassigned
          8.
          		 Adaptive shuffle merge finalization for push-based shuffle Sub-task Resolved Venkata krishnan Sowrirajan
          9.
          		 Improve caching of MergeStatus on the executor side to save memory Sub-task In Progress Unassigned
          10.
          		 Adaptive fetch of shuffle mergers for Push based shuffle Sub-task Resolved Venkata krishnan Sowrirajan
          11.
          		 Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition Sub-task Open Unassigned
          12.
          		 When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size. Sub-task Open Unassigned
          13.
          		 Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time. Sub-task Open Unassigned
          14.
          		 Support push based shuffle when barrier scheduling is enabled Sub-task Open Unassigned
          15.
          		 Handling new attempt has started error message in BlockPushErrorHandler in client Sub-task Resolved Qi Zhu
          16.
          		 Register merge status even after shuffle dependency is merge finalized Sub-task Open Unassigned
          17.
          		 Cleanup RemoteBlockPushResolver log messages Sub-task Resolved Venkata krishnan Sowrirajan
          18.
          		 Fix some typos in ShuffleBlockPusher class. Sub-task Resolved Qi Zhu
          19.
          		 Minor changes to address a few identified server side inefficiencies Sub-task Resolved Min Shen
          20.
          		 Randomize blocks within a push request before pushing to improve block merge ratio Sub-task Resolved Min Shen
          21.
          		 Pull out NoOpMergedShuffleFileManager inner class outside Sub-task Resolved Venkata krishnan Sowrirajan
          22.
          		 Handle Stale block fetch failure on the client side by not retrying the requests Sub-task Resolved Unassigned
          23.
          		 Client side related push-based shuffle metrics Sub-task Resolved Thejdeep Gudivada
          24.
          		 Support IO encryption for push-based shuffle Sub-task Open Unassigned
          25.
          		 Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry Sub-task Resolved Minchu Yang
          26.
          		 Child stage using merged output or not should be based on the availability of merged output from parent stage Sub-task Open Unassigned
          27.
          		 Prevent overwriting of push shuffle merged files once the shuffle is finalized Sub-task Resolved Chandni Singh
          28.
          		 PushBlockStreamCallback should check isTooLate first to avoid NPE Sub-task Resolved Cheng Pan
          29.
          		 Replace usages of slaveTracker to workerTracker in MapOutputTrackerSuite Sub-task Open Unassigned
          30.
          		 Support cleaning up merged shuffle files and state from external shuffle service Sub-task Resolved Unassigned
          31.
          		 Push-based shuffle's internal implementation details should not be exposed as API Sub-task Open Unassigned
          32.
          		 Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus Sub-task Open Unassigned
          33.
          		 Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized Sub-task Open Unassigned
          34.
          		 Fallback to original shuffle block when a push-merged shuffle chunk is zero-size Sub-task Resolved gaoyajun02
          35.
          		 JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              csingh Chandni Singh
              Votes:
              1 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated: