[YARN-149] [Umbrella] ResourceManager (RM) Fail-over

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: resourcemanager
• Labels: patch
• Target Version/s: 2.4.0
• Assignee: Unassigned
• Reporter: Harsh J
• Created: 17/Jun/12 07:07
• Updated: 03/May/15 01:50
• Resolved: 03/May/15 01:49

Description

This jira tracks work needed to be done to support one RM instance failing over to another RM instance so that we can have RM HA. Work includes leader election, transfer of control to leader and client re-direction to new leader.

Issue Links

• duplicates: MAPREDUCE-225 Fault tolerant Hadoop Job Tracker
• is blocked by: YARN-1318 Promote AdminService to an Always-On service and merge in RMHAProtocolService
• is part of: MAPREDUCE-279 Map-Reduce 2.0
• is related to: HADOOP-9905 remove dependency of zookeeper for hadoop-client
• is related to: MAPREDUCE-2288 JT Availability
• is related to: YARN-128 [Umbrella] RM Restart Phase 1: State storage and non-work-preserving recovery
• is related to: YARN-556 [Umbrella] RM Restart phase 2 - Work preserving restart
• relates to: YARN-1305 RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
• relates to: YARN-1139 [Umbrella] Convert all RM components to Services
• relates to: YARN-1543 Stateless multiple resource manager
• relates to: YARN-1460 Revisit YARN client connection configs

Sub-Tasks

• Separate out RM services into "Always On" and "Active": Closed
• Implement RMHAProtocolService: Closed
• Test and verify ACL based ZKRMStateStore fencing for RM State Store: Resolved
• Add FailoverProxyProvider like capability to RMProxy: Closed
• Allow embedding leader election into the RM: Closed
• Expose RM active/standby state to Web UI and REST API: Closed
• Add admin support for HA operations: Closed
• Revisit exception handling in ZKRMStateStore post RM HA: Resolved
• Add shutdown support to non-service RM components: Open
• Support automatic failover using ZKFC: Open
• Add end-to-end tests for HA: Open
• Move init() of activeServices to ResourceManager#serviceStart(): Resolved
• Augment MiniYARNCluster to support HA mode: Closed
• Update HAServiceState to STOPPING on RM#stop(): Resolved
• ResourceManger.clusterTimeStamp should be reset when RM transitions to active: Resolved
• Verify RM HA works in secure clusters: Resolved
• Make improvements in ZKRMStateStore for fencing: Closed
• RM DT token service should have service addresses of both RMs: Closed
• Configuration to support multiple RMs: Closed
• RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException: Closed
• Promote AdminService to an Always-On service and merge in RMHAProtocolService: Closed
• Set HTTPS webapp address along with other RPC addresses in HAUtil: Closed
• Enabling HA should check Configuration contains multiple RMs: Closed
• RM should log using RMStore at startup time: Closed
• Handle RM fails over after getApplicationID() and before submitApplication().: Closed
• HA config shouldn't affect NodeManager RPC addresses: Closed
• RM services should depend on ConfigurationProvider during startup too: Closed
• Move internal services logic from AdminService to ResourceManager: Closed
• WebApplicationProxy should be always-on w.r.t HA even if it is embedded in the RM: Closed
• Enabling HA should verify the RM service addresses configurations have been set for every RM Ids defined in RM_HA_IDs: Closed
• Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA: Closed
• Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation: Closed
• Use StandbyException instead of RMNotYetReadyException: Closed
• Web UI should redirect to active RM when HA is enabled.: Closed
• Add an option to yarn rmadmin to clear the znode used by embedded elector: Open
• Add an easy way to turn on HA: Resolved
• Race between ServerRMProxy and ClientRMProxy setting RMProxy#INSTANCE: Closed
• ZK store should use a private password for root-node-acls: Closed
• RMDispatcher should be reset on transition to standby: Closed
• ActiveRMInfoProto fields should be optional: Closed
• Support explicit failover when automatic failover is enabled: Resolved
• HA-related rmadmin commands don't work on a secure cluster: Closed
• Make admin refresh of capacity scheduler configuration work across RM failover: Closed
• YARM RM HA requires different configs on different RM hosts: Closed
• Manual Failover does not work in secure clusters: Closed
• ZK store should attempt a write periodically to ensure it is still Active: Closed
• RMDTRenewer#getRMClient should use ClientRMProxy: Closed
• Webservice should redirect to active RM when HA is enabled.: Closed
• add the ability to set yarn.resourcemanager.hostname.rm-id instead of setting all the various host:port properties for RM: Closed
• Set better defaults for HA configs for automatic failover: Closed
• Make admin refreshNodes work across RM failover: Closed
• Make admin refreshSuperUserGroupsConfiguration work across RM failover: Closed
• Make admin refreshAdminAcls work across RM failover: Closed
• Make admin refreshServiceAcls work across RM failover: Closed
• Make admin refreshUserToGroupsMappings of configuration work across RM failover: Closed
• Make admin refresh of Fair scheduler configuration work across RM failover: Open
• Cleanup YARN HAUtil class: Open
• Document RM HA: Closed
• Reset cluster-metrics on transition to standby: Closed
• RM should get the updated Configurations when it transits from Standby to Active: Closed
• RMAdminCLI should check whether HA is enabled before executes transitionToActive/transitionToStandby: Closed
• Handle RM failovers during the submitApplication call.: Resolved
• Handle RM fail overs after the submitApplication call.: Closed
• Write test cases to verify that killApplication API works in RM HA: Closed
• When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.: Closed
• renewDelegationToken should survive RM failover: Closed
• Handle AMRMTokens across RM failover: Closed
• RM HA: AM link broken if the AM is on nodes other than RM: Closed
• Add retry cache support in ResourceManager: Resolved
• Persist ClusterMetrics across RM HA transitions: Open
• cancelDelegationToken should survive RM failover: Open
• Both RM stuck in standby mode when automatic failover is enabled: Closed
• Document yarn.resourcemanager.zk-auth and its scope: Closed
• Implement and verify Scheduler#moveApplication() idempotent for CapacityScheduler/FairScheduler: Open
• Make ApplicationMasterProtocol#allocate AtMostOnce: Closed
• Add testcases to test AMRMToken on HA: Resolved
• Standby RM's conf, stacks, logLevel, metrics, jmx and logs links are redirecting to Active RM: Closed
• Yarn standby RM taking long to transition to active: Open
• Aggregation of MR job logs failing when Resourcemanager switches: Resolved
• NM-Local dir cleanup failing when Resourcemanager switches: Open
• Option "--forceactive" not works as described in usage of "yarn rmadmin -transitionToActive": Closed
• [RM HA] Rest api endpoints doing redirect incorrectly: Closed
• Improve the error message when attempting manual failover with auto-failover enabled: Closed
• forcemanual transitionToStandby in RM-HA automatic-failover mode should change elector state: Open
• Documentation of ResourceManager HA should explain configurations about listen addresses: Closed
• Both RM in active state when Admin#transitionToActive failure from refeshAll(): Closed
• RM HA UI redirection needs to be fixed when both RMs are in standby mode: Closed
• RM should print alert messages if Zookeeper and Resourcemanager gets connection issue: Closed
• Both RM becomes Active if all zookeepers can not connect to active RM: Resolved
• Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit: Resolved