[SPARK-41469] Task rerun on decommissioned executor can be avoided if shuffle data has migrated

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.2.2, 3.3.1
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: wuyi
• Reporter: wuyi
• Created: 09/Dec/22 15:13
• Updated: 02/Sep/24 02:20
• Resolved: 27/Dec/22 21:38

Description

Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.

Issue Links

• is related to: SPARK-49472 Resubmit the task on executor decommission
• links to: [Github] Pull Request #39011 (Ngone51)
• links to: GitHub Pull Request #39011