Bulk Operation

SPARK-33235 JsonProtocol should skip logging of push-based shuffle read metrics when push-based shuffle is disabled

SPARK-33235 Fallback to original shuffle block when a push-merged shuffle chunk is zero-size

SPARK-33235 Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized

SPARK-33235 Check if shuffleMergeId is the same as the current stage's shuffleMergeId before registering MergeStatus

SPARK-33235 Push-based shuffle's internal implementation details should not be exposed as API

SPARK-33235 Support cleaning up merged shuffle files and state from external shuffle service

SPARK-33235 Replace usages of slaveTracker to workerTracker in MapOutputTrackerSuite

SPARK-33235 PushBlockStreamCallback should check isTooLate first to avoid NPE

SPARK-33235 Prevent overwriting of push shuffle merged files once the shuffle is finalized

SPARK-33235 Child stage using merged output or not should be based on the availability of merged output from parent stage

SPARK-33235 Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

SPARK-33235 Support IO encryption for push-based shuffle

SPARK-33235 Client side related push-based shuffle metrics

SPARK-33235 Handle Stale block fetch failure on the client side by not retrying the requests

SPARK-33235 Pull out NoOpMergedShuffleFileManager inner class outside

SPARK-33235 Randomize blocks within a push request before pushing to improve block merge ratio

SPARK-33235 Minor changes to address a few identified server side inefficiencies

SPARK-33235 Fix some typos in ShuffleBlockPusher class.

SPARK-33235 Cleanup RemoteBlockPushResolver log messages

SPARK-33235 Register merge status even after shuffle dependency is merge finalized

SPARK-33235 Handling new attempt has started error message in BlockPushErrorHandler in client

SPARK-33235 Support push based shuffle when barrier scheduling is enabled

SPARK-33235 Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.

SPARK-33235 When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size.

SPARK-33235 Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition

SPARK-33235 Adaptive fetch of shuffle mergers for Push based shuffle

SPARK-33235 Improve caching of MergeStatus on the executor side to save memory

SPARK-33235 Adaptive shuffle merge finalization for push-based shuffle

SPARK-33235 Enable ShuffleBlockPusher to stop pushing blocks for a particular shuffle partition

SPARK-33235 Improve locality for push-based shuffle especially for join like operations

SPARK-33235 Server side metrics related to push-based shuffle

SPARK-33235 Better heuristics to compute number of shuffle mergers required for a ShuffleMapStage

SPARK-33235 Limit the number of pending blocks in memory and store blocks that collide

SPARK-33235 Pluggable API to fetch shuffle merger locations with Push based shuffle

SPARK-33235 Enable Push-based shuffle service to store state in NM level DB for work preserving restart