[SPARK-32738] thread safe endpoints may hang due to fatal error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.4, 2.4.6, 3.0.0
• Fix Version/s: 2.4.8, 3.0.2, 3.1.0
• Component/s: Spark Core
• Assignee: Zhenhua Wang
• Reporter: Zhenhua Wang
• Created: 30/Aug/20 04:20
• Updated: 31/Aug/21 08:18
• Resolved: 15/Sep/20 06:46

Description

Processing for `ThreadSafeRpcEndpoint` is controlled by 'numActiveThreads' in `Inbox`. Now if any fatal error happens during `Inbox.process`, 'numActiveThreads' is not reduced. Then other threads can not process messages in that inbox, which causes the endpoint to "hang".

This problem is more serious in previous Spark 2.x versions since the driver, executor and block manager endpoints are all thread safe endpoints.

Issue Links

• links to: [Github] Pull Request #29580 (wzhfy)
• links to: [Github] Pull Request #29763 (wzhfy)
• links to: [Github] Pull Request #29763 (wzhfy)
• links to: [Github] Pull Request #29764 (wzhfy)
• links to: [Github] Pull Request #29764 (wzhfy)
• links to: [Github] Pull Request #33872 (sleep1661)
• links to: [Github] Pull Request #33872 (sleep1661)