Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7453

Fix issue where RM fails to switch to active after first successful start

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.1.0
    • Fix Version/s: 2.9.0, 3.0.0, 3.1.0
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      It is observed that RM fail to switch to ACTIVE after first successful start! The below exception throws when RM is switching from ACTIVE->STANDBY->ACTIVE. This continues in loop!

      2017-11-07 15:08:11,664 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to active state
      2017-11-07 15:08:11,669 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recovery started
      2017-11-07 15:08:11,669 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Loaded RM state version info 1.5
      2017-11-07 15:08:11,670 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Failed to load/recover state
      org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
      	at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
      	at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
      	at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
      	at org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
      	at org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
      	at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
      	at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
      	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
      	at org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
      	at org.apache.hadoop.util.curator.ZKCuratorManager$SafeTransaction.commit(ZKCuratorManager.java:403)
      	at org.apache.hadoop.util.curator.ZKCuratorManager.safeSetData(ZKCuratorManager.java:372)
      	at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:493)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:771)
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1162)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1202)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1198)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1198)
      	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
      	at org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
      	at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:894)
      	at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473)
      	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:607)
      	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
      

        Attachments

        1. YARN-7453.001.patch
          10 kB
          Sunil Govindan
        2. YARN-7453.001.patch
          5 kB
          Rohith Sharma K S

          Activity

            People

            • Assignee:
              rohithsharma Rohith Sharma K S
              Reporter:
              rohithsharma Rohith Sharma K S
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: