Bulk Operation

SPARK-30602 Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

SPARK-30602 Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

SPARK-30602 Stage has all tasks finished but with ongoing finalization can cause job hang

SPARK-30602 Avoid finalizing when there's no push at all in a shuffle

SPARK-30602 Push-based shuffle documentation

SPARK-30602 Rename classes in shuffle RPC used for block push operations

SPARK-30602 FileNotFoundException from the shuffle push can cause the executor to terminate

SPARK-30602 Disable push-based shuffle until the feature is complete

SPARK-30602 Add Support in the ESS to serve merged shuffle block meta and data to executors

SPARK-30602 Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

SPARK-30602 Fix cases of corruption in merged shuffle blocks that are pushed

SPARK-30602 Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

SPARK-30602 Add support to properly handle different type of stage retries

SPARK-30602 Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures

SPARK-30602 Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario

SPARK-30602 Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage

SPARK-30602 Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions

SPARK-30602 RPC implementation to support control plane coordination for push-based shuffle

SPARK-30602 Add support for executors to push shuffle blocks after successful map task completion

SPARK-30602 Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle

SPARK-30602 RPC implementation to support pushing and merging shuffle blocks