[YARN-149] [Umbrella] ResourceManager (RM) Fail-over

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: resourcemanager
• Labels: patch
• Target Version/s: 2.4.0
• Assignee: Unassigned
• Reporter: Harsh J
• Created: 17/Jun/12 07:07
• Updated: 03/May/15 01:50
• Resolved: 03/May/15 01:49

Description

This jira tracks work needed to be done to support one RM instance failing over to another RM instance so that we can have RM HA. Work includes leader election, transfer of control to leader and client re-direction to new leader.

Issue Links

• duplicates: MAPREDUCE-225 Fault tolerant Hadoop Job Tracker
• is blocked by: YARN-1318 Promote AdminService to an Always-On service and merge in RMHAProtocolService
• is part of: MAPREDUCE-279 Map-Reduce 2.0
• is related to: HADOOP-9905 remove dependency of zookeeper for hadoop-client
• is related to: MAPREDUCE-2288 JT Availability
• is related to: YARN-128 [Umbrella] RM Restart Phase 1: State storage and non-work-preserving recovery
• is related to: YARN-556 [Umbrella] RM Restart phase 2 - Work preserving restart
• relates to: YARN-1305 RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
• relates to: YARN-1139 [Umbrella] Convert all RM components to Services
• relates to: YARN-1543 Stateless multiple resource manager
• relates to: YARN-1460 Revisit YARN client connection configs

Sub-Tasks

• Add shutdown support to non-service RM components: Open
• Support automatic failover using ZKFC: Open
• Add end-to-end tests for HA: Open
• Add an option to yarn rmadmin to clear the znode used by embedded elector: Open
• Make admin refresh of Fair scheduler configuration work across RM failover: Open
• Cleanup YARN HAUtil class: Open
• Persist ClusterMetrics across RM HA transitions: Open
• cancelDelegationToken should survive RM failover: Open
• Implement and verify Scheduler#moveApplication() idempotent for CapacityScheduler/FairScheduler: Open
• Yarn standby RM taking long to transition to active: Open
• NM-Local dir cleanup failing when Resourcemanager switches: Open
• forcemanual transitionToStandby in RM-HA automatic-failover mode should change elector state: Open