[SPARK-4694] Long-run user thread(such as HiveThriftServer2) causes the 'process leak' in yarn-client mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: YARN
• Labels: backport-needed
• Target Version/s: 1.2.1, 1.3.0
• Assignee: Unassigned
• Reporter: Zhaowei Huang
• Created: 02/Dec/14 12:56
• Updated: 29/Dec/14 22:13
• Resolved: 17/Dec/14 20:24

Description

Recently when I use the Yarn HA mode to test the HiveThriftServer2 I found a problem that the driver can't exit by itself.
To reappear it, you can do as fellow:
1.use yarn HA mode and set am.maxAttemp = 1for convenience
2.kill the active resouce manager in cluster

The expect result is just failed, because the maxAttemp was 1.

But the actual result is that: all executor was ended but the driver was still there and never close.

Issue Links

• links to: [Github] Pull Request #3576 (SaintBacchus)