Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4127

RM fail with noAuth error if switched from failover mode to non-failover mode

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The scenario is that RM failover was initially enabled, so the zkRootNodeAcl is by default set with the RM ID in the ACL string

      If RM failover is then switched to be disabled, it cannot load data from ZK and fail with noAuth error. After I reset the root node ACL, it again can access.

      15/09/08 14:28:34 ERROR resourcemanager.ResourceManager: Failed to load/recover state
      org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
        at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:949)
        at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
        at org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
        at org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
        at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
        at org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
        at org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$SafeTransaction.commit(ZKRMStateStore.java:1009)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.safeSetData(ZKRMStateStore.java:985)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:374)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:579)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:973)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1014)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1010)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1010)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1050)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1194)
      

      the problem may be that in non-failover mode, RM doesn't use the RM-ID to connect with ZK and thus fail with no Auth error.

      We should be able to switch failover on and off with no interruption to the user.

      1. YARN-4127.01.patch
        6 kB
        Varun Saxena
      2. YARN-4127.02.patch
        6 kB
        Varun Saxena
      3. YARN-4127-branch-2.7.01.patch
        8 kB
        Varun Saxena
      4. YARN-4127-branch-2.7.02.patch
        8 kB
        Varun Saxena

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Jian He for the review and commit

        Show
        varun_saxena Varun Saxena added a comment - Thanks Jian He for the review and commit
        Hide
        jianhe Jian He added a comment -

        Committed branch-2.7 patch, thanks Varun Saxena !

        Show
        jianhe Jian He added a comment - Committed branch-2.7 patch, thanks Varun Saxena !
        Hide
        varun_saxena Varun Saxena added a comment -

        Filed YARN-4328 for the findbugs issue reported above

        Show
        varun_saxena Varun Saxena added a comment - Filed YARN-4328 for the findbugs issue reported above
        Hide
        varun_saxena Varun Saxena added a comment -

        Test failures are unrelated. Except the one related to node labels, there are JIRAs' corresponding to them. Anyways all are unrelated.
        Findbugs is also unrelated. Will raise a separate JIRA for it.
        Whitespace issues seem unrelated too.

        Show
        varun_saxena Varun Saxena added a comment - Test failures are unrelated. Except the one related to node labels, there are JIRAs' corresponding to them. Anyways all are unrelated. Findbugs is also unrelated. Will raise a separate JIRA for it. Whitespace issues seem unrelated too.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 43s branch-2.7 passed
        +1 compile 0m 22s branch-2.7 passed with JDK v1.8.0_60
        +1 compile 0m 23s branch-2.7 passed with JDK v1.7.0_79
        +1 checkstyle 0m 18s branch-2.7 passed
        +1 mvneclipse 0m 18s branch-2.7 passed
        -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 cannot run convertXmlToText from findbugs
        +1 javadoc 0m 23s branch-2.7 passed with JDK v1.8.0_60
        +1 javadoc 0m 27s branch-2.7 passed with JDK v1.7.0_79
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 21s the patch passed with JDK v1.8.0_60
        +1 javac 0m 21s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_79
        +1 javac 0m 23s the patch passed
        -1 checkstyle 0m 13s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 226, now 229).
        +1 mvneclipse 0m 14s the patch passed
        -1 whitespace 0m 0s The patch has 1549 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 39s The patch has 137 line(s) with tabs.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_60
        +1 javadoc 0m 24s the patch passed with JDK v1.7.0_79
        -1 unit 51m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60.
        -1 unit 52m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79.
        -1 asflicense 44m 58s Patch generated 65 ASF License warnings.
        165m 36s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770248/YARN-4127-branch-2.7.02.patch
        JIRA Issue YARN-4127
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux 8b6165a9a169 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
        git revision branch-2.7 / ece1b16
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9618/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 197MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9618/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 43s branch-2.7 passed +1 compile 0m 22s branch-2.7 passed with JDK v1.8.0_60 +1 compile 0m 23s branch-2.7 passed with JDK v1.7.0_79 +1 checkstyle 0m 18s branch-2.7 passed +1 mvneclipse 0m 18s branch-2.7 passed -1 findbugs 1m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 cannot run convertXmlToText from findbugs +1 javadoc 0m 23s branch-2.7 passed with JDK v1.8.0_60 +1 javadoc 0m 27s branch-2.7 passed with JDK v1.7.0_79 +1 mvninstall 0m 25s the patch passed +1 compile 0m 21s the patch passed with JDK v1.8.0_60 +1 javac 0m 21s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_79 +1 javac 0m 23s the patch passed -1 checkstyle 0m 13s Patch generated 4 new checkstyle issues in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager (total was 226, now 229). +1 mvneclipse 0m 14s the patch passed -1 whitespace 0m 0s The patch has 1549 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 39s The patch has 137 line(s) with tabs. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_79 -1 unit 51m 18s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60. -1 unit 52m 21s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79. -1 asflicense 44m 58s Patch generated 65 ASF License warnings. 165m 36s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-03 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770248/YARN-4127-branch-2.7.02.patch JIRA Issue YARN-4127 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux 8b6165a9a169 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision branch-2.7 / ece1b16 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9618/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9618/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 197MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9618/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Jian He, updated branch-2.7 patch

        Show
        varun_saxena Varun Saxena added a comment - Jian He , updated branch-2.7 patch
        Hide
        jianhe Jian He added a comment -

        Varun Saxena, I committed yarn-4321. This patch has come conflicts, mind updating please? thanks !

        Show
        jianhe Jian He added a comment - Varun Saxena , I committed yarn-4321. This patch has come conflicts, mind updating please? thanks !
        Hide
        varun_saxena Varun Saxena added a comment -

        Jian He, raised YARN-4321 for this issue.

        Show
        varun_saxena Varun Saxena added a comment - Jian He , raised YARN-4321 for this issue.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2491 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2491/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2491 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2491/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Hide
        varun_saxena Varun Saxena added a comment -

        Jian He

        however for the branch-2.7 patch, if I run the test case without the core change, the test will keep in a loop and not finish. could you take a look ?

        This is because we do not handle NoAuth exception properly in branch-2.7 code when HA is not enabled.
        In ZKRMStateStore#runWithRetries, we have code as under. As can be seen if HA is not enabled, we neither rethrow NoAuthException nor do we have any logic increment retries and back out if retries are maxed out.
        With fix in this patch, probably NoAuth will never come until and unless someone changes it from CLI. I will go ahead and file another JIRA.

            T runWithRetries() throws Exception {
              int retry = 0;
              while (true) {
                try {
                  return runWithCheck();
                } catch (KeeperException.NoAuthException nae) {
                  if (HAUtil.isHAEnabled(getConfig())) {
                    // NoAuthException possibly means that this store is fenced due to
                    // another RM becoming active. Even if not,
                    // it is safer to assume we have been fenced
                    throw new StoreFencedException();
                  }
                } catch (KeeperException ke) {
                  .............
               }
             }
          }
        
        Show
        varun_saxena Varun Saxena added a comment - Jian He however for the branch-2.7 patch, if I run the test case without the core change, the test will keep in a loop and not finish. could you take a look ? This is because we do not handle NoAuth exception properly in branch-2.7 code when HA is not enabled. In ZKRMStateStore#runWithRetries, we have code as under. As can be seen if HA is not enabled, we neither rethrow NoAuthException nor do we have any logic increment retries and back out if retries are maxed out. With fix in this patch, probably NoAuth will never come until and unless someone changes it from CLI. I will go ahead and file another JIRA. T runWithRetries() throws Exception { int retry = 0; while ( true ) { try { return runWithCheck(); } catch (KeeperException.NoAuthException nae) { if (HAUtil.isHAEnabled(getConfig())) { // NoAuthException possibly means that this store is fenced due to // another RM becoming active. Even if not, // it is safer to assume we have been fenced throw new StoreFencedException(); } } catch (KeeperException ke) { ............. } } }
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #553 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/553/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #553 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/553/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1339 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1339/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1339 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1339/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2546/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2546 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2546/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #604 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/604/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #604 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/604/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/615/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/615/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8727 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8727/)
        YARN-4127. RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8727 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8727/ ) YARN-4127 . RM fail with noAuth error if switched from failover to (jianhe: rev e5b1733e049dc0f1859b93618354e049a0efdc4a) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
        Hide
        jianhe Jian He added a comment -

        Varun Saxena, I committed the patch into trunk and branch-2,
        however for the branch-2.7 patch, if I run the test case without the core change, the test will keep in a loop and not finish. could you take a look ? thanks !

        Show
        jianhe Jian He added a comment - Varun Saxena , I committed the patch into trunk and branch-2, however for the branch-2.7 patch, if I run the test case without the core change, the test will keep in a loop and not finish. could you take a look ? thanks !
        Hide
        varun_saxena Varun Saxena added a comment -

        Jian He, kindly review

        Show
        varun_saxena Varun Saxena added a comment - Jian He , kindly review
        Hide
        varun_saxena Varun Saxena added a comment -

        This failure is due to branch-2.7 patch(as QA tries to apply it on trunk). The QA report for patch on trunk is above and fine.

        Show
        varun_saxena Varun Saxena added a comment - This failure is due to branch-2.7 patch(as QA tries to apply it on trunk). The QA report for patch on trunk is above and fine.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768217/YARN-4127-branch-2.7.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 124a412
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9539/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768217/YARN-4127-branch-2.7.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 Console output https://builds.apache.org/job/PreCommit-YARN-Build/9539/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        In branch-2.7, error message is as under :

        2015-10-23 09:22:10,209 DEBUG [SyncThread:0] server.DataTree (DataTree.java:processTxn(949)) - Ignoring processTxn failure hdr: -1 : error: -102
        2015-10-23 09:22:10,210 DEBUG [main-SendThread(127.0.0.1:11221)] zookeeper.ClientCnxn (ClientCnxn.java:readResponse(818)) - Reading reply sessionid:0x15092d1ebe10001, packet:: clientPath:null serverPath:null finished:false header:: 7591,1  replyHeader:: 7591,7610,-102  request:: '/rmstore/ZKRMStateRoot/RMAppRoot,,v{s{31,s{'world,'anyone}}},0  response::
        2015-10-23 09:22:10,210 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x15092d1ebe10001 type:create cxid:0x1da8 zxid:0x1dbb txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth
        
        Show
        varun_saxena Varun Saxena added a comment - In branch-2.7, error message is as under : 2015-10-23 09:22:10,209 DEBUG [SyncThread:0] server.DataTree (DataTree.java:processTxn(949)) - Ignoring processTxn failure hdr: -1 : error: -102 2015-10-23 09:22:10,210 DEBUG [main-SendThread(127.0.0.1:11221)] zookeeper.ClientCnxn (ClientCnxn.java:readResponse(818)) - Reading reply sessionid:0x15092d1ebe10001, packet:: clientPath:null serverPath:null finished:false header:: 7591,1 replyHeader:: 7591,7610,-102 request:: '/rmstore/ZKRMStateRoot/RMAppRoot,,v{s{31,s{'world,'anyone}}},0 response:: 2015-10-23 09:22:10,210 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x15092d1ebe10001 type:create cxid:0x1da8 zxid:0x1dbb txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 8s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 53s There were no new javac warning messages.
        +1 javadoc 10m 27s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 51s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 58m 9s Tests passed in hadoop-yarn-server-resourcemanager.
            98m 32s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12768191/YARN-4127.02.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 124a412
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9537/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9537/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9537/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 javadoc 10m 27s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 58m 9s Tests passed in hadoop-yarn-server-resourcemanager.     98m 32s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12768191/YARN-4127.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 124a412 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9537/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9537/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9537/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Jian He, updated the patch. The patch does not apply cleanly on branch-2.7 so will update a patch for that too.

        Show
        varun_saxena Varun Saxena added a comment - Jian He , updated the patch. The patch does not apply cleanly on branch-2.7 so will update a patch for that too.
        Hide
        varun_saxena Varun Saxena added a comment -

        Sorry Jian He, missed your earlier comment.
        Makes sense to delete fencing node path. Will update a patch shortly.

        Show
        varun_saxena Varun Saxena added a comment - Sorry Jian He , missed your earlier comment. Makes sense to delete fencing node path. Will update a patch shortly.
        Hide
        jianhe Jian He added a comment -

        Hi Varun Saxena, any updates?

        Show
        jianhe Jian He added a comment - Hi Varun Saxena , any updates?
        Hide
        jianhe Jian He added a comment -

        thanks Varun ! patch looks good to me overall,

        delete(fencingNodePath); 

        do you think this can also be called if HA is disabled so that the fencingNodePath is also deleted if switched from HA to non-HA.

        Show
        jianhe Jian He added a comment - thanks Varun ! patch looks good to me overall, delete(fencingNodePath); do you think this can also be called if HA is disabled so that the fencingNodePath is also deleted if switched from HA to non-HA.
        Hide
        varun_saxena Varun Saxena added a comment -

        All the test failures are unrelated. They are due to no class def found and hence must be because of parallel builds.

        Show
        varun_saxena Varun Saxena added a comment - All the test failures are unrelated. They are due to no class def found and hence must be because of parallel builds.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 14s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 9s There were no new javac warning messages.
        +1 javadoc 10m 22s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 51s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 59m 23s Tests failed in hadoop-yarn-server-resourcemanager.
            100m 20s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestApplicationMasterService
          hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12762528/YARN-4127.01.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 861b52d
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9275/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9275/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9275/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 9s There were no new javac warning messages. +1 javadoc 10m 22s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 51s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 59m 23s Tests failed in hadoop-yarn-server-resourcemanager.     100m 20s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestApplicationMasterService   hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762528/YARN-4127.01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 861b52d hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9275/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9275/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9275/console This message was automatically generated.

          People

          • Assignee:
            varun_saxena Varun Saxena
            Reporter:
            jianhe Jian He
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development