[SPARK-29298] Separate block manager heartbeat endpoint from driver endpoint

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Lantao Jin
• Reporter: Lantao Jin
• Created: 30/Sep/19 03:35
• Updated: 12/Nov/19 08:25
• Resolved: 12/Nov/19 08:25

Description

Executor's heartbeat will send synchronously to BlockManagerMaster to let it know that the block manager is still alive. In a heavy cluster, it will timeout and cause block manager re-register unexpected.
This improvement will separate a heartbeat endpoint from the driver endpoint. In our production environment, this is really helpful to prevent executors from unstable up and down.

Issue Links

• links to: GitHub Pull Request #25971