[YARN-2992] ZKRMStateStore crashes due to session expiry

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Component/s: resourcemanager
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Hadoop Flags: Reviewed
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 25/Dec/14 22:26
• Updated: 30/Aug/16 01:28
• Resolved: 27/Dec/14 02:01

Description

We recently saw the RM crash with the following stacktrace. On session expiry, we should gracefully transition to standby.

2014-12-18 06:28:42,689 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type STATE_STORE_OP_FAILED. Cause: 
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired 
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) 
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:931) 
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:911) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:930) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$4.run(ZKRMStateStore.java:927) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1069) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1088) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:927) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.doMultiWithRetries(ZKRMStateStore.java:941) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.setDataWithRetries(ZKRMStateStore.java:958) 
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:687)