[SPARK-41469] Task rerun on decommissioned executor can be avoided if shuffle data has migrated

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.2.2, 3.3.1
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: wuyi
• Reporter: wuyi
• Created: 09/Dec/22 15:13
• Updated: 27/Dec/22 21:38
• Resolved: 27/Dec/22 21:38

Description

Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.

Issue Links

• links to: [Github] Pull Request #39011 (Ngone51)