Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9797

Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      RequestHedgingProxyProvider tries to connect to all the Namenodes and reports the exception for all the standby exceptions for all the other namenodes. There is no point on reporting the standby exception if it's expected.

      1. HDFS-9797-v000.patch
        3 kB
        Inigo Goiri
      2. HDFS-9797-v001.patch
        3 kB
        Inigo Goiri

        Issue Links

          Activity

          Hide
          elgoiri Inigo Goiri added a comment -

          With HDFS-6440, now we can have more than two NNs. With the current RequestHedgingProxyProvider, if we have 4 NNs, we see:
          Invocation returned exception on [hostA]
          Invocation returned exception on [hostC]
          Invocation returned exception on [hostD]

          Those exception are usually caused by trying to query the Standby NN. Given that this is the expected behavior, I propose to check the exception and just show a debugging message for those.

          Show
          elgoiri Inigo Goiri added a comment - With HDFS-6440 , now we can have more than two NNs. With the current RequestHedgingProxyProvider , if we have 4 NNs, we see: Invocation returned exception on [hostA] Invocation returned exception on [hostC] Invocation returned exception on [hostD] Those exception are usually caused by trying to query the Standby NN. Given that this is the expected behavior, I propose to check the exception and just show a debugging message for those.
          Hide
          elgoiri Inigo Goiri added a comment -

          Proposal to check if it is caused by an standby exception and just show as debug.

          Show
          elgoiri Inigo Goiri added a comment - Proposal to check if it is caused by an standby exception and just show as debug.
          Hide
          asuresh Arun Suresh added a comment -

          Makes Sense.. Thanks for raising this Inigo Goiri !!
          The patch looks mostly good. Very minor nits :

          1. Remove the ret boolean variable in the isStandbyException method, replace the inner most if with return (unwrapRemoteException instanceof StandbyException) and do a return false; in the end.
          2. Maybe pull lines 124 - 132 into a new private method log(ex, tProxyInfo.proxyInfo)

          +1 pending the above

          Show
          asuresh Arun Suresh added a comment - Makes Sense.. Thanks for raising this Inigo Goiri !! The patch looks mostly good. Very minor nits : Remove the ret boolean variable in the isStandbyException method, replace the inner most if with return (unwrapRemoteException instanceof StandbyException) and do a return false; in the end. Maybe pull lines 124 - 132 into a new private method log(ex, tProxyInfo.proxyInfo) +1 pending the above
          Hide
          elgoiri Inigo Goiri added a comment -

          Tackling Arun Suresh's comments.

          Show
          elgoiri Inigo Goiri added a comment - Tackling Arun Suresh 's comments.
          Hide
          elgoiri Inigo Goiri added a comment -

          Tackling Arun Suresh's comments

          Show
          elgoiri Inigo Goiri added a comment - Tackling Arun Suresh 's comments
          Hide
          elgoiri Inigo Goiri added a comment -

          Tackling Arun Suresh comments (and refixing javadoc issues, even though they are in private methods).

          Show
          elgoiri Inigo Goiri added a comment - Tackling Arun Suresh comments (and refixing javadoc issues, even though they are in private methods).
          Hide
          elgoiri Inigo Goiri added a comment -

          Thanks Arun Suresh for the feedback, I attached an updated version of the patch.
          Let me know if you have any comments on v001.

          I think jenkins should trigger a build now; let's see if it comes clean.

          Show
          elgoiri Inigo Goiri added a comment - Thanks Arun Suresh for the feedback, I attached an updated version of the patch. Let me know if you have any comments on v001. I think jenkins should trigger a build now; let's see if it comes clean.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 22s trunk passed
          +1 compile 0m 38s trunk passed with JDK v1.8.0_72
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 3s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 43s the patch passed
          +1 compile 0m 36s the patch passed with JDK v1.8.0_72
          +1 javac 0m 36s the patch passed
          +1 compile 0m 38s the patch passed with JDK v1.7.0_95
          +1 javac 0m 38s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95
          -1 unit 51m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 49m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          124m 0s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDatanodeRegistration
            hadoop.fs.TestHdfsNativeCodeLoader
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.fs.TestHdfsNativeCodeLoader



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787613/HDFS-9797-v000.patch
          JIRA Issue HDFS-9797
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 74c9e594c15a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9b959f3
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14473/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 77MB
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14473/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 22s trunk passed +1 compile 0m 38s trunk passed with JDK v1.8.0_72 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 3s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 43s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_72 +1 javac 0m 36s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95 -1 unit 51m 34s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 49m 12s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 124m 0s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestDatanodeRegistration   hadoop.fs.TestHdfsNativeCodeLoader JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.fs.TestHdfsNativeCodeLoader Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787613/HDFS-9797-v000.patch JIRA Issue HDFS-9797 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 74c9e594c15a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9b959f3 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14473/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14473/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 77MB Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14473/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          It looks like the test case failures are unrelated and this being a logging improvement, is not easily testable via unit tests.
          Inigo Goiri, Can you confirm that it works as expected during manual tests ?
          Will commit after that..

          Show
          asuresh Arun Suresh added a comment - It looks like the test case failures are unrelated and this being a logging improvement, is not easily testable via unit tests. Inigo Goiri , Can you confirm that it works as expected during manual tests ? Will commit after that..
          Hide
          elgoiri Inigo Goiri added a comment -

          Yes, I tested with a few thousand containers and they don't show the Standby exceptions and only the regular exceptions.

          Show
          elgoiri Inigo Goiri added a comment - Yes, I tested with a few thousand containers and they don't show the Standby exceptions and only the regular exceptions.
          Hide
          asuresh Arun Suresh added a comment -

          Committed to trunk, branch-2 and branch-2.8

          Show
          asuresh Arun Suresh added a comment - Committed to trunk, branch-2 and branch-2.8
          Hide
          elgoiri Inigo Goiri added a comment -

          Thank you Arun Suresh very much for the review and the commit!

          Show
          elgoiri Inigo Goiri added a comment - Thank you Arun Suresh very much for the review and the commit!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9297 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9297/)
          HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider (arun suresh: rev 9fdfb546fb67526ba261da5cbd005f33e0f1d9e1)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9297 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9297/ ) HDFS-9797 . Log Standby exceptions thrown by RequestHedgingProxyProvider (arun suresh: rev 9fdfb546fb67526ba261da5cbd005f33e0f1d9e1) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 11s trunk passed
          +1 compile 0m 42s trunk passed with JDK v1.8.0_66
          +1 compile 0m 43s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.8.0_66
          +1 javac 0m 42s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.7.0_91
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 19s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91
          -1 unit 63m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          +1 unit 57m 48s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          148m 8s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.TestFileAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787628/HDFS-9797-v001.patch
          JIRA Issue HDFS-9797
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f37cc8d928f9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e6a7044
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14479/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14479/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14479/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14479/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 11s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 55s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 49s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_91 +1 javac 0m 41s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_91 -1 unit 63m 13s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 57m 48s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 148m 8s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.TestFileAppend Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12787628/HDFS-9797-v001.patch JIRA Issue HDFS-9797 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f37cc8d928f9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e6a7044 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14479/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14479/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14479/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14479/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              elgoiri Inigo Goiri
              Reporter:
              elgoiri Inigo Goiri
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development