Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
ShuffleBlockPusher, which was introduced in SPARK-32917 stops pushing shuffle blocks for the entire shuffle when it receives "TOO Late" exception. However, with the change https://github.com/apache/spark/pull/30433, there is also a need to stop pushing shuffle blocks for a particular reduce partition. Refer https://github.com/apache/spark/pull/30433#discussion_r533694433