[YARN-2884] Proxying all AM-RM communications

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.9.0, 3.0.0-alpha1
• Component/s: nodemanager, resourcemanager
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Kishore Chaliparambil
• Reporter: Carlo Curino
• Created: 20/Nov/14 19:00
• Updated: 25/Oct/19 20:27
• Resolved: 08/Sep/15 03:25

Description

We introduce the notion of an RMProxy, running on each node (or once per rack). Upon start the AM is forced (via tokens and configuration) to direct all its requests to a new services running on the NM that provide a proxy to the central RM.

This give us a place to:
1) perform distributed scheduling decisions
2) throttling mis-behaving AMs
3) mask the access to a federation of RMs

Issue Links

• blocks: YARN-3666 Federation Intercepting and propagating AM- home RM communications
• depends upon: YARN-4083 Add a discovery mechanism for the scheduler address
• depends upon: YARN-4117 End to end unit test with mini YARN cluster for AMRMProxy Service
• is blocked by: YARN-3116 [Collector wireup] We need an assured way to determine if a container is an AM container on NM
• is blocked by: YARN-3927 Make the NodeManager's ContainerManager pluggable
• relates to: YARN-5411 Create a proxy chain for ApplicationClientProtocol in the Router
• relates to: YARN-5833 Add validation to ensure default ports are unique in Configuration
• relates to: YARN-1547 Prevent DoS of ApplicationMasterProtocol by putting in limits
• relates to: YARN-2915 Enable YARN RM scale out via federation using multiple RM's
• relates to: YARN-6355 [Umbrella] Preprocessor framework for AM and Client interactions with the RM