[YARN-2915] Enable YARN RM scale out via federation using multiple RM's

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: nodemanager, resourcemanager
• Labels: federation
• Hadoop Flags: Reviewed

Release Note

A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.

• Assignee: Subramaniam Krishnan
• Reporter: Sriram Rao
• Created: 02/Dec/14 17:07
• Updated: 05/Mar/19 01:44
• Resolved: 25/Sep/17 21:10

Description

This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising of tens of thousands of nodes. That is, rather than limiting a YARN managed cluster to about 4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.

Issue Links

• blocks: YARN-6848 Move Router ClientRMServices Interceptor and chain into yarn api and common package
• depends upon: YARN-4879 Enhance Allocate Protocol to Identify Requests Explicitly
• is blocked by: YETUS-485 Yetus run is failing on branch after rebase/force push
• is depended upon by: REEF-337 Support REEF on YARN Federation
• is depended upon by: YARN-5597 YARN Federation improvements
• is related to: YARN-5357 Timeline service v2 integration with Federation
• is related to: YARN-2884 Proxying all AM-RM communications
• is related to: HADOOP-12427 [JDK8] Upgrade Mockito version to 1.10.19
• relates to: HADOOP-13378 Common features between YARN and HDFS Router-based federation
• relates to: HDFS-10467 Router-based HDFS federation
• relates to: REEF-568 Work around the federated YARN node reports problem
• relates to: REEF-589 REEF crashes when new nodes are added to the clusters dynamically

Sub-Tasks

• Federation Membership State Store internal APIs: Resolved
• Federation State and Policy Store (DBMS implementation): Resolved
• Federation PolicyStore internal APIs: Resolved
• Federation subcluster membership mechanisms: Resolved
• Federation Intercepting and propagating AM- home RM communications: Resolved
• Federation: transparently spanning application across multiple sub-clusters: Resolved
• Integrate Federation services with ResourceManager: Resolved
• Create Facade for Federation State and Policy Store: Resolved
• Create a FailoverProxy for Federation services: Resolved
• Add a flag in container to indicate whether it's an AM container or not: Resolved
• Make the NodeManager's ContainerManager pluggable: Resolved
• Exclude generated federation protobuf sources from YARN Javadoc/findbugs build: Resolved
• Federation Application State Store internal APIs: Resolved
• Policies APIs (for Router and AMRMProxy policies): Resolved
• Stateless Federation router policies implementation: Resolved
• Stateless ARMRMProxy policies implementation: Resolved
• PolicyManager to tie together Router/AMRM Federation policies: Resolved
• Simplify initialization/use of RouterPolicy via a RouterPolicyFacade: Resolved
• Federation Subcluster Resolver: Resolved
• In-memory based implementation of the FederationMembershipStateStore: Resolved
• In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore: Resolved
• Compose Federation membership/application/policy APIs into an uber FederationStateStore API: Resolved
• Bootstrap Router server module: Resolved
• Federation: routing client invocations transparently to multiple RMs: Resolved
• Create a proxy chain for ApplicationClientProtocol in the Router: Resolved
• Create a proxy chain for ResourceManager REST API in the Router: Resolved
• Create a proxy chain for ResourceManager Admin API in the Router: Resolved
• InputValidator for the FederationStateStore internal APIs: Resolved
• Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover: Resolved
• UnmanagedAM pool manager for federating application across clusters: Resolved
• Make the RM epoch base value configurable: Resolved
• Utils for Federation State and Policy Store: Resolved
• Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover: Resolved
• Add a HashBasedRouterPolicy, and small policies and test refactoring.: Resolved
• Refactor TestPBImplRecords so that we can reuse for testing protocol records in other YARN modules: Resolved
• Add AlwayReject policies for router and amrmproxy.: Resolved
• Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP: Resolved
• Add support for work preserving NM restart when AMRMProxy is enabled: Resolved
• Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy: Resolved
• Occasional test failure in TestWeightedRandomRouterPolicy: Resolved
• Fix minor bugs in handling of local AMRMToken in AMRMProxy: Resolved
• Support multiple attempts on the node when AMRMProxy is enabled: Resolved
• Share a single instance of SubClusterResolver instead of instantiating one per AM: Resolved
• Cleanup when AMRMProxy fails to initialize a new interceptor chain: Resolved
• Recreate interceptor chain for different attemptId in the same node in AMRMProxy: Resolved
• [Documentation] Documenting the YARN Federation feature: Resolved
• [Regression] TestFederationRMStateStoreService is failing with null pointer exception: Resolved
• Fix memory leak and finish app trigger in AMRMProxy: Resolved
• Refactor of ResourceManager#startWebApp in a Util class: Resolved
• Fix unit test failure in TestRouterClientRMService: Resolved
• Add ability to blacklist sub-clusters when invoking Routing policies: Resolved
• Adding required missing configs to Federation configuration guide based on e2e testing: Resolved
• [Bug] FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not: Resolved
• Move FederationStateStore SQL DDL files from test resource to sbin: Resolved
• Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk: Resolved
• Refactoring RMWebServices by moving some util methods to RMWebAppUtil: Resolved
• Add MySql Scripts for FederationStateStore: Resolved
• Update Microsoft JDBC Driver for SQL Server version in License.txt: Resolved
• Handle concurrent register AM requests in FederationInterceptor: Resolved
• Add PoolInitializationException as retriable exception in FederationFacade: Resolved
• Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution): Resolved
• ZooKeeper based implementation of the FederationStateStore: Resolved
• Metrics for Federation StateStore: Resolved
• Metrics for Federation Router: Resolved
• Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps): Resolved
• Federation: routing getNode/getNodes/getMetrics REST invocations transparently to multiple RMs: Resolved
• Update YARN daemon startup/shutdown scripts to include Router service: Resolved
• Federation: routing ClientRM invocations transparently to multiple RMs (part 2 - getApps): Resolved
• Federation: routing ClientRM invocations transparently to multiple RMs (part 5 - getNode/getNodes/getMetrics): Resolved
• Add support for updateContainers when allocating using FederationInterceptor: Resolved
• Basic Federation UI: Resolved