[SPARK-39957] Delay onDisconnected to enable Driver receives ExecutorExitCode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Kai-Hsun Chen
• Reporter: Kai-Hsun Chen
• Created: 03/Aug/22 01:19
• Updated: 24/Aug/22 12:50
• Resolved: 24/Aug/22 12:50

Description

There are two methods to detect executor loss. First, when RPC fails, the function onDisconnected will be triggered. Second, when executor exits with ExecutorExitCode, the exit code will be passed from ExecutorRunner to Driver. These two methods may categorize same cases into different conclusions. We hope to categorize the ExecutorLossReason by ExecutorExitCode. This PR aims to make sure Driver receives ExecutorExitCode before onDisconnected is called.

Issue Links

• links to: [Github] Pull Request #37400 (kevin85421)
• links to: [Github] Pull Request #37400 (kevin85421)