[SPARK-3591] Provide "fire and forget" option for YARN cluster mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.4.0
• Component/s: YARN
• Target Version/s: 1.4.0
• Assignee: Tao Wang (Inactive)
• Reporter: Andrew Or
• Created: 18/Sep/14 17:18
• Updated: 07/Apr/15 13:36
• Resolved: 07/Apr/15 13:36

Description

After launching an application through yarn-cluster mode, the SparkSubmit process sticks around and enters a monitoring loop to track the application's status. This is really a responsibility that belongs to a different process, such that SparkSubmit can run yarn-cluster applications in a fire-and-forget mode. We currently already do this for standalone-cluster mode. we should do it for yarn-cluster mode too.

Issue Links

• is related to: SPARK-6692 Add an option for client to kill AM when it is killed
• links to: [Github] Pull Request #5297 (WangTaoTheTonic)