[SPARK-40481] Ignore stage fetch failure caused by decommissioned executor

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Zhongwei Zhu
• Reporter: Zhongwei Zhu
• Created: 18/Sep/22 21:54
• Updated: 16/Dec/22 18:54
• Resolved: 03/Oct/22 16:42

Description

When executor decommission is enabled, there would be many stage failure caused by FetchFailed from decommissioned executor, further causing whole job's failure. It would be better not to count such failure in `spark.stage.maxConsecutiveAttempts`

Issue Links

• links to: [Github] Pull Request #37924 (warrenzhu25)
• links to: [Github] Pull Request #37924 (warrenzhu25)
• links to: [Github] Pull Request #38441 (warrenzhu25)
• links to: [Github] Pull Request #38441 (warrenzhu25)