[SPARK-34245] Master may not remove the finished executor when Worker fails to send ExecutorStateChanged

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.7, 3.0.1, 3.1.1, 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Deploy, Spark Core
• Assignee: wuyi
• Reporter: wuyi
• Created: 26/Jan/21 15:04
• Updated: 24/Feb/21 05:05
• Resolved: 24/Feb/21 05:05

Description

If the Worker fails to send ExecutorStateChanged to the Master due to some errors, e.g., temporary network error, then the Master can't remove the finished executor normally and think the executor is still alive. In the worst case, if the executor is the only one executor for the application, the application can get hang.

 

Issue Links

• links to: [Github] Pull Request #31348 (Ngone51)