[YARN-128] [Umbrella] RM Restart Phase 1: State storage and non-work-preserving recovery

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Component/s: resourcemanager
• Assignee: Unassigned
• Reporter: Arun Murthy
• Created: 08/Jun/12 04:05
• Updated: 08/May/15 18:07
• Resolved: 03/May/15 02:29

Description

This umbrella jira tracks the work needed to preserve critical state information and reload them upon RM restart.

Issue Links

• depends upon: MAPREDUCE-5086 MR app master deletes staging dir when sent a reboot command from the RM
• is blocked by: MAPREDUCE-5505 Clients should be notified job finished only after job successfully unregistered
• is blocked by: YARN-1082 Secure RM with recovery enabled and rm state store on hdfs fails with gss exception
• is blocked by: YARN-495 Change NM behavior of reboot to resync
• is related to: MAPREDUCE-5476 Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM
• relates to: MAPREDUCE-5471 Succeed job tries to restart after RMrestart
• relates to: MAPREDUCE-5127 MR job succeeds and exits even when unregister with RM fails
• relates to: MAPREDUCE-5466 Historyserver does not refresh the result of restarted jobs after RM restart
• relates to: YARN-1305 RMHAProtocolService#serviceInit should handle HAUtil's IllegalArgumentException
• relates to: MAPREDUCE-5472 reducer of sort job restarts from scratch in between after RM restart
• relates to: MAPREDUCE-5567 [Umbrella] Stabilize MR framework w.r.t ResourceManager restart
• relates to: YARN-209 Capacity scheduler doesn't trigger app-activation after adding nodes
• relates to: YARN-479 NM retry behavior for connection to RM should be similar for lost heartbeats
• relates to: YARN-149 [Umbrella] ResourceManager (RM) Fail-over
• relates to: YARN-218 Distiguish between "failed" and "killed" app attempts
• relates to: YARN-556 [Umbrella] RM Restart phase 2 - Work preserving restart
• relates to: YARN-1139 [Umbrella] Convert all RM components to Services

Sub-Tasks

• Remove old code for restart: Closed
• Make changes for RM restart phase 1: Closed
• Add FS-based persistent store implementation for RMStateStore: Closed
• Add FileSystem based store for RM: Resolved
• Security related work for RM restart: Resolved
• Add Zookeeper-based store implementation for RMStateStore: Closed
• Add HDFS based store for RM which manages the store using directories: Resolved
• Create common proxy client for communicating with RM: Closed
• Delayed store operations should not result in RM unavailability for app submission: Closed
• AM max attempts is not checked when RM restart and try to recover attempts: Closed
• Race condition causing RM to potentially relaunch already unregistered AMs on RM restart: Closed
• NM should reject containers allocated by previous RM: Closed
• Test and verify that app delegation tokens are added to tokenRenewer after RM restart: Closed
• Restore appToken and clientToken for app attempt after RM restart: Closed
• Restore clientToken for app attempt after RM restart: Resolved
• Restore RMDelegationTokens after RM Restart: Closed
• RMStateStore's removeApplication APIs should just take an applicationId: Resolved
• Slow or failing DelegationToken renewals on submission itself make RM unavailable: Closed
• verify that new jobs submitted with old RM delegation tokens after RM restart are accepted: Closed
• Store completed application information in RM state store: Closed
• RM crashes if it restarts while the state-store is down: Closed
• Apps Completed metrics on web UI is not correct after RM restart: Resolved
• List of applications at NM web UI is inconsistent with applications at RM UI after RM restart: Resolved
• Change FileSystemRMStateStore to use directories: Closed
• Document RM Restart feature: Closed
• "Active users" field in Resourcemanager scheduler UI gives negative values: Resolved
• Handle app recovery differently for AM failures and RM restart: Resolved
• Recovery issues on RM Restart with FileSystemRMStateStore: Resolved
• Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts: Closed
• RMStateStore should flush all pending store events before closing: Closed
• RM may relaunch already KILLED / FAILED jobs after RM restarts: Resolved
• AM fails to register if RM restarts within 5s of job submission: Resolved
• During RM restart, RM should start a new attempt only when previous attempt exits for real: Closed
• Register ClientToken MasterKey in SecretManager after it is saved: Closed
• Save version information in the state store: Closed
• FileSystemRMStateStore can leave partial files that prevent subsequent recovery: Closed
• Rethink znode structure for RM HA: Closed
• Batching optimization for ZKRMStateStore: Resolved
• Implement a RMStateStore cleaner for deleting application/attempt info: Closed
• RM hangs on shutdown if calling system.exit in serviceInit or serviceStart: Closed
• Check time cost for recovering max-app-limit applications: Resolved
• Change killing application to wait until state store is done: Closed
• Apps should be saved after it's accepted by the scheduler: Open
• Fix invalid RMApp transition from NEW to FINAL_SAVING: Closed
• Revisit RMApp transitions from NEW on RECOVER: Resolved
• Execessive logging for app and attempts on RM recovery: Open
• Work preserving recovery of Unmanged AMs: Resolved
• NPE on registerNodeManager if the request has containers for UnmanagedAMs: Closed
• Job stays in PREP state for long time after RM Restarts: Closed
• Succeeded application remains in accepted after RM restart: Closed
• Better reporting of finished containers to AMs: Resolved
• RM should honor NM heartbeat expiry after RM restart: Open
• Move RM recovery related proto to yarn_server_resourcemanager_recovery.proto: Closed
• Remove ApplicationAttemptState and ApplicationState class in RMStateStore class: Closed
• Add leveldb-based implementation for RMStateStore: Closed
• RMProxy should retry EOFException: Closed