Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10341

Add a metric to expose the timeout number of pending replication blocks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Per HDFS-6682, recording the timeout number of pending replication blocks is useful to get the cluster health.

      1. HDFS-10341.04.patch
        7 kB
        Akira Ajisaka
      2. HDFS-10341.03.patch
        7 kB
        Akira Ajisaka
      3. HDFS-10341.02.patch
        6 kB
        Akira Ajisaka
      4. HDFS-10341.01.patch
        7 kB
        Akira Ajisaka
      5. HDFS-10341.01.branch-2.patch
        7 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          ajisakaa Akira Ajisaka added a comment -

          Findbugs warning is also unrelated to the patch. Filed HDFS-10485 to fix it.

          Show
          ajisakaa Akira Ajisaka added a comment - Findbugs warning is also unrelated to the patch. Filed HDFS-10485 to fix it.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed to branch-2 and branch-2.8. Thanks Arpit Agarwal for the quick review!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed to branch-2 and branch-2.8. Thanks Arpit Agarwal for the quick review!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Whitespace issues and test failures seem to be unrelated to the patch. Committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - Whitespace issues and test failures seem to be unrelated to the patch. Committing this.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 31s Maven dependency ordering for branch
          +1 mvninstall 6m 49s branch-2 passed
          +1 compile 6m 29s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 49s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 22s branch-2 passed
          +1 mvnsite 1m 47s branch-2 passed
          +1 mvneclipse 0m 25s branch-2 passed
          -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings.
          +1 javadoc 2m 11s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 2m 57s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 30s the patch passed
          +1 compile 6m 29s the patch passed with JDK v1.8.0_91
          +1 javac 6m 29s the patch passed
          +1 compile 6m 38s the patch passed with JDK v1.7.0_101
          +1 javac 6m 38s the patch passed
          +1 checkstyle 1m 30s the patch passed
          +1 mvnsite 1m 54s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 4m 9s the patch passed
          +1 javadoc 2m 11s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 56s the patch passed with JDK v1.7.0_101
          +1 unit 9m 39s hadoop-common in the patch passed with JDK v1.8.0_91.
          -1 unit 63m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          +1 unit 8m 57s hadoop-common in the patch passed with JDK v1.7.0_101.
          -1 unit 60m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_101.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          205m 38s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
            hadoop.hdfs.TestAsyncHDFSWithHA
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          JDK v1.7.0_101 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestAsyncHDFSWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807882/HDFS-10341.01.branch-2.patch
          JIRA Issue HDFS-10341
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux d3a383431696 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / cddf6b4
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15639/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15639/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 31s Maven dependency ordering for branch +1 mvninstall 6m 49s branch-2 passed +1 compile 6m 29s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 49s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 22s branch-2 passed +1 mvnsite 1m 47s branch-2 passed +1 mvneclipse 0m 25s branch-2 passed -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 extant Findbugs warnings. +1 javadoc 2m 11s branch-2 passed with JDK v1.8.0_91 +1 javadoc 2m 57s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 6m 29s the patch passed with JDK v1.8.0_91 +1 javac 6m 29s the patch passed +1 compile 6m 38s the patch passed with JDK v1.7.0_101 +1 javac 6m 38s the patch passed +1 checkstyle 1m 30s the patch passed +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 4m 9s the patch passed +1 javadoc 2m 11s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 56s the patch passed with JDK v1.7.0_101 +1 unit 9m 39s hadoop-common in the patch passed with JDK v1.8.0_91. -1 unit 63m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 8m 57s hadoop-common in the patch passed with JDK v1.7.0_101. -1 unit 60m 42s hadoop-hdfs in the patch failed with JDK v1.7.0_101. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 205m 38s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength   hadoop.hdfs.TestAsyncHDFSWithHA   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs JDK v1.7.0_101 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestAsyncHDFSWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807882/HDFS-10341.01.branch-2.patch JIRA Issue HDFS-10341 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux d3a383431696 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / cddf6b4 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15639/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15639/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the branch-2 patch also, thanks Akira Ajisaka.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the branch-2 patch also, thanks Akira Ajisaka .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Xiaobing for reviewing and thanks Arpit for reviewing & committing. Attached a patch for branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Xiaobing for reviewing and thanks Arpit for reviewing & committing. Attached a patch for branch-2.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Thank you Arpit Agarwal for committing. Akira Ajisaka would you please also provide a patch for branch-2? Thanks.

          Show
          xiaobingo Xiaobing Zhou added a comment - Thank you Arpit Agarwal for committing. Akira Ajisaka would you please also provide a patch for branch-2? Thanks.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9903/)
          HDFS-10341. Add a metric to expose the timeout number of pending (arp: rev 97e244947719d483c3f80521a00fec8e13dcb637)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReconstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9903/ ) HDFS-10341 . Add a metric to expose the timeout number of pending (arp: rev 97e244947719d483c3f80521a00fec8e13dcb637) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReconstructionBlocks.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReconstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 for the v4 patch. The test failures are unrelated. I pushed the fix to trunk. Thanks Akira Ajisaka and thanks Xiaobing Zhou for the review.

          The branch-2 patch will need changes due to refactoring introduced by HDFS-9869.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 for the v4 patch. The test failures are unrelated. I pushed the fix to trunk. Thanks Akira Ajisaka and thanks Xiaobing Zhou for the review. The branch-2 patch will need changes due to refactoring introduced by HDFS-9869 .
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Akira Ajisaka patch v04 looks good. Can you verify if the test failures are related to the patch? Thank you.

          Show
          xiaobingo Xiaobing Zhou added a comment - Akira Ajisaka patch v04 looks good. Can you verify if the test failures are related to the patch? Thank you.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 32s Maven dependency ordering for branch
          +1 mvninstall 6m 16s trunk passed
          +1 compile 6m 9s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 1m 41s trunk passed
          +1 mvneclipse 0m 23s trunk passed
          +1 findbugs 2m 52s trunk passed
          +1 javadoc 1m 57s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 6m 9s the patch passed
          +1 javac 6m 9s the patch passed
          +1 checkstyle 1m 22s the patch passed
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 29s the patch passed
          +1 javadoc 2m 9s the patch passed
          -1 unit 8m 13s hadoop-common in the patch failed.
          -1 unit 72m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          120m 55s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807388/HDFS-10341.04.patch
          JIRA Issue HDFS-10341
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux e2e78ac6526b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d749cf6
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15621/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15621/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 32s Maven dependency ordering for branch +1 mvninstall 6m 16s trunk passed +1 compile 6m 9s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 23s trunk passed +1 findbugs 2m 52s trunk passed +1 javadoc 1m 57s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 6m 9s the patch passed +1 javac 6m 9s the patch passed +1 checkstyle 1m 22s the patch passed +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 29s the patch passed +1 javadoc 2m 9s the patch passed -1 unit 8m 13s hadoop-common in the patch failed. -1 unit 72m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 120m 55s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807388/HDFS-10341.04.patch JIRA Issue HDFS-10341 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux e2e78ac6526b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d749cf6 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15621/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15621/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15621/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Arpit Agarwal and Xiaobing Zhou for the comments. Updated the patch to address the comment.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Arpit Agarwal and Xiaobing Zhou for the comments. Updated the patch to address the comment.
          Hide
          xiaobingo Xiaobing Zhou added a comment -

          Akira Ajisaka thank you for the work. Would you like to post a new patch to address Arpit Agarwal's comments?

          Show
          xiaobingo Xiaobing Zhou added a comment - Akira Ajisaka thank you for the work. Would you like to post a new patch to address Arpit Agarwal 's comments?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Akira Ajisaka, that makes sense. So IIUC the metric may count the same block multiple times as timed out blocks are reinserted into the needed replications queue. If so perhaps we should rename the metric to NumTimedOutPendingReconstructions and update the documentation to state that it counts the number of timed out reconstructions and not the number of unique blocks that timed out?

          +1 with those updates.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Akira Ajisaka , that makes sense. So IIUC the metric may count the same block multiple times as timed out blocks are reinserted into the needed replications queue. If so perhaps we should rename the metric to NumTimedOutPendingReconstructions and update the documentation to state that it counts the number of timed out reconstructions and not the number of unique blocks that timed out? +1 with those updates.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 7m 24s trunk passed
          +1 compile 7m 39s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 2m 22s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 20s trunk passed
          +1 javadoc 2m 24s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 57s the patch passed
          +1 compile 7m 54s the patch passed
          +1 javac 7m 54s the patch passed
          -1 checkstyle 1m 42s root: patch generated 1 new + 334 unchanged - 0 fixed = 335 total (was 334)
          +1 mvnsite 1m 41s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 43s the patch passed
          +1 javadoc 2m 9s the patch passed
          +1 unit 8m 41s hadoop-common in the patch passed.
          +1 unit 72m 26s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          127m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806014/HDFS-10341.02.patch
          JIRA Issue HDFS-10341
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 2900f0a9aa38 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / edd716e
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15551/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15551/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15551/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 24s trunk passed +1 compile 7m 39s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 2m 22s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 20s trunk passed +1 javadoc 2m 24s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 57s the patch passed +1 compile 7m 54s the patch passed +1 javac 7m 54s the patch passed -1 checkstyle 1m 42s root: patch generated 1 new + 334 unchanged - 0 fixed = 335 total (was 334) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 43s the patch passed +1 javadoc 2m 9s the patch passed +1 unit 8m 41s hadoop-common in the patch passed. +1 unit 72m 26s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 127m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806014/HDFS-10341.02.patch JIRA Issue HDFS-10341 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 2900f0a9aa38 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / edd716e Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15551/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15551/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15551/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          03: Expose the total number of timed out pending replication blocks without using additional AtomicInteger.

          Show
          ajisakaa Akira Ajisaka added a comment - 03: Expose the total number of timed out pending replication blocks without using additional AtomicInteger.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Arpit for the comment. Updated the patch.

          I wanted to expose the total number of timeouts, so I created additional AtomicInt in the previous patch. In the 02 patch, the metric shows the current number of timed out pending replication blocks because timedOutItems is cleared in getTimedOutBlocks().
          I suspect that the current number doesn't fit for us because the number is cleared very frequently (recheck interval is 3 sec by default, which is probably smaller than the interval of metrics sink). What do you think?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Arpit for the comment. Updated the patch. I wanted to expose the total number of timeouts, so I created additional AtomicInt in the previous patch. In the 02 patch, the metric shows the current number of timed out pending replication blocks because timedOutItems is cleared in getTimedOutBlocks() . I suspect that the current number doesn't fit for us because the number is cleared very frequently (recheck interval is 3 sec by default, which is probably smaller than the interval of metrics sink). What do you think?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Akira Ajisaka, can we just return timedOutItems.size() by querying it within the timedOutItems object lock instead of adding a new Atomic int?

          I checked every place that lock is taken and it is fairly lightweight so so I see no problem acquiring it when querying metrics. Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Akira Ajisaka , can we just return timedOutItems.size() by querying it within the timedOutItems object lock instead of adding a new Atomic int? I checked every place that lock is taken and it is fairly lightweight so so I see no problem acquiring it when querying metrics. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 10m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 21s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 6m 8s trunk passed with JDK v1.8.0_91
          +1 compile 7m 9s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 13s trunk passed
          +1 mvnsite 1m 54s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 34s trunk passed
          +1 javadoc 2m 5s trunk passed with JDK v1.8.0_91
          +1 javadoc 2m 54s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 6m 14s the patch passed with JDK v1.8.0_91
          +1 javac 6m 15s the patch passed
          +1 compile 6m 53s the patch passed with JDK v1.7.0_95
          +1 javac 6m 53s the patch passed
          -1 checkstyle 1m 12s root: patch generated 1 new + 321 unchanged - 0 fixed = 322 total (was 321)
          +1 mvnsite 1m 54s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 11s the patch passed
          +1 javadoc 2m 3s the patch passed with JDK v1.8.0_91
          +1 javadoc 2m 55s the patch passed with JDK v1.7.0_95
          +1 unit 7m 42s hadoop-common in the patch passed with JDK v1.8.0_91.
          -1 unit 62m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_91.
          +1 unit 8m 14s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 59m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          210m 58s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.tools.TestDFSZKFailoverController
            hadoop.hdfs.TestRollingUpgrade
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:7b1c37a
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801194/HDFS-10341.01.patch
          JIRA Issue HDFS-10341
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 089350440c16 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6f26b66
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15314/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15314/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 21s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 6m 8s trunk passed with JDK v1.8.0_91 +1 compile 7m 9s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 13s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 34s trunk passed +1 javadoc 2m 5s trunk passed with JDK v1.8.0_91 +1 javadoc 2m 54s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 6m 14s the patch passed with JDK v1.8.0_91 +1 javac 6m 15s the patch passed +1 compile 6m 53s the patch passed with JDK v1.7.0_95 +1 javac 6m 53s the patch passed -1 checkstyle 1m 12s root: patch generated 1 new + 321 unchanged - 0 fixed = 322 total (was 321) +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 11s the patch passed +1 javadoc 2m 3s the patch passed with JDK v1.8.0_91 +1 javadoc 2m 55s the patch passed with JDK v1.7.0_95 +1 unit 7m 42s hadoop-common in the patch passed with JDK v1.8.0_91. -1 unit 62m 9s hadoop-hdfs in the patch failed with JDK v1.8.0_91. +1 unit 8m 14s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 59m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 210m 58s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:7b1c37a JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801194/HDFS-10341.01.patch JIRA Issue HDFS-10341 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 089350440c16 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6f26b66 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HDFS-Build/15314/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15314/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15314/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              ajisakaa Akira Ajisaka
              Reporter:
              ajisakaa Akira Ajisaka
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development