Description
With push-based shuffle enabled, the reducers send 2 different requests to the ESS:
1. Request to fetch the metadata of the merged shuffle block.
2. Requests to fetch the data of the merged shuffle block in chunks which are by default 2MB in size.
The ESS should be able to serve these requests and this Jira targets all the changes in the network-common and network-shuffle modules to be able to support this.
Attachments
Issue Links
- Dependent
-
SPARK-32922 Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
-
- Resolved
-
- links to
