Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4321

Incessant retries if NoAuthException is thrown by Zookeeper in non HA mode

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.7.2
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This applies to only branch-2.7 or earlier code.
      When a NoAuthException is thrown in non HA mode(like in the scenario of YARN-4127), RM incessantly keeps on retrying the ZK operation.

      2015-10-23 09:22:10,209 DEBUG [SyncThread:0] server.DataTree (DataTree.java:processTxn(949)) - Ignoring processTxn failure hdr: -1 : error: -102
      2015-10-23 09:22:10,210 DEBUG [main-SendThread(127.0.0.1:11221)] zookeeper.ClientCnxn (ClientCnxn.java:readResponse(818)) - Reading reply sessionid:0x15092d1ebe10001, packet:: clientPath:null serverPath:null finished:false header:: 7591,1  replyHeader:: 7591,7610,-102  request:: '/rmstore/ZKRMStateRoot/RMAppRoot,,v{s{31,s{'world,'anyone}}},0  response::
      2015-10-23 09:22:10,210 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(645)) - Got user-level KeeperException when processing sessionid:0x15092d1ebe10001 type:create cxid:0x1da8 zxid:0x1dbb txntype:-1 reqpath:n/a Error Path:null Error:KeeperErrorCode = NoAuth
      

      This is because we do not handle NoAuthException properly in branch-2.7 code when HA is not enabled.
      In ZKRMStateStore#runWithRetries, we have code as under. As can be seen if HA is not enabled, we neither rethrow NoAuthException nor do we have any logic to increment retries and back out if retries are maxed out.

       T runWithRetries() throws Exception {
            int retry = 0;
            while (true) {
              try {
                return runWithCheck();
              } catch (KeeperException.NoAuthException nae) {
                if (HAUtil.isHAEnabled(getConfig())) {
                  // NoAuthException possibly means that this store is fenced due to
                  // another RM becoming active. Even if not,
                  // it is safer to assume we have been fenced
                  throw new StoreFencedException();
                }
              } catch (KeeperException ke) {
                .............
             }
           }
        }
      

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Jian He for the review and commit

        Show
        varun_saxena Varun Saxena added a comment - Thanks Jian He for the review and commit
        Hide
        jianhe Jian He added a comment -

        Committed to branch-2.7 thanks Varun Saxena !

        Show
        jianhe Jian He added a comment - Committed to branch-2.7 thanks Varun Saxena !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        -1 mvninstall 8m 2s root in branch-2.7 failed.
        +1 compile 0m 20s branch-2.7 passed with JDK v1.8.0_60
        +1 compile 0m 22s branch-2.7 passed with JDK v1.7.0_79
        +1 checkstyle 0m 19s branch-2.7 passed
        +1 mvneclipse 0m 19s branch-2.7 passed
        -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 cannot run convertXmlToText from findbugs
        +1 javadoc 0m 24s branch-2.7 passed with JDK v1.8.0_60
        +1 javadoc 0m 26s branch-2.7 passed with JDK v1.7.0_79
        +1 mvninstall 0m 24s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.8.0_60
        +1 javac 0m 22s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_79
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        -1 whitespace 0m 0s The patch has 886 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 23s The patch has 95 line(s) with tabs.
        +1 findbugs 1m 15s the patch passed
        +1 javadoc 0m 19s the patch passed with JDK v1.8.0_60
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_79
        -1 unit 50m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60.
        -1 unit 52m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79.
        -1 asflicense 40m 41s Patch generated 67 ASF License warnings.
        160m 39s



        Reason Tests
        JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769828/YARN-4321-branch-2.7.01.patch
        JIRA Issue YARN-4321
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux f991bae132cd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh
        git revision branch-2.7 / 7b78391
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9606/testReport/
        asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 195MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9606/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. -1 mvninstall 8m 2s root in branch-2.7 failed. +1 compile 0m 20s branch-2.7 passed with JDK v1.8.0_60 +1 compile 0m 22s branch-2.7 passed with JDK v1.7.0_79 +1 checkstyle 0m 19s branch-2.7 passed +1 mvneclipse 0m 19s branch-2.7 passed -1 findbugs 1m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 cannot run convertXmlToText from findbugs +1 javadoc 0m 24s branch-2.7 passed with JDK v1.8.0_60 +1 javadoc 0m 26s branch-2.7 passed with JDK v1.7.0_79 +1 mvninstall 0m 24s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_60 +1 javac 0m 22s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_79 +1 javac 0m 23s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvneclipse 0m 15s the patch passed -1 whitespace 0m 0s The patch has 886 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 23s The patch has 95 line(s) with tabs. +1 findbugs 1m 15s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_60 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_79 -1 unit 50m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_60. -1 unit 52m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_79. -1 asflicense 40m 41s Patch generated 67 ASF License warnings. 160m 39s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_79 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-30 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769828/YARN-4321-branch-2.7.01.patch JIRA Issue YARN-4321 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux f991bae132cd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-e77b1ce/precommit/personality/hadoop.sh git revision branch-2.7 / 7b78391 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9606/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/9606/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 195MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9606/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Straightforward fix.
        I think we dont need to retry for NoAuthException as the exception is unlikely to change even after retries.

        Show
        varun_saxena Varun Saxena added a comment - Straightforward fix. I think we dont need to retry for NoAuthException as the exception is unlikely to change even after retries.

          People

          • Assignee:
            varun_saxena Varun Saxena
            Reporter:
            varun_saxena Varun Saxena
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development