[SPARK-3822] Expose a mechanism for SparkContext to ask for / remove Yarn containers

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.2.0
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 06/Oct/14 20:40
• Updated: 29/Oct/14 21:01
• Resolved: 29/Oct/14 21:01

Description

This is one of the core components for the umbrella issue SPARK-3174. Currently, the only agent in Spark that communicates directly with the RM is the AM. This means the only way for the SparkContext to ask for / remove containers from the RM is through the AM. The communication link between the SparkContext and the AM needs to be added.

Issue Links

• links to: [Github] Pull Request #2798 (PraveenSeluka)
• links to: [Github] Pull Request #2840 (andrewor14)