Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12131

Add some of the FSNamesystem JMX values as metrics

    Details

    • Hadoop Flags:
      Reviewed

      Description

      A number of useful numbers are emitted via the FSNamesystem JMX, but not through the metrics system. These would be useful to be able to track over time, e.g. to alert on via standard metrics systems or to view trends and rate changes:

      • NumLiveDataNodes
      • NumDeadDataNodes
      • NumDecomLiveDataNodes
      • NumDecomDeadDataNodes
      • NumDecommissioningDataNodes
      • NumStaleStorages
      • VolumeFailuresTotal
      • EstimatedCapacityLostTotal
      • NumInMaintenanceLiveDataNodes
      • NumInMaintenanceDeadDataNodes
      • NumEnteringMaintenanceDataNodes

      This is a simple change that just requires annotating the JMX methods with @Metric.

      1. HDFS-12131.000.patch
        2 kB
        Erik Krogen
      2. HDFS-12131.001.patch
        8 kB
        Erik Krogen
      3. HDFS-12131.002.patch
        10 kB
        Erik Krogen
      4. HDFS-12131.002.patch
        10 kB
        Erik Krogen
      5. HDFS-12131.003.patch
        15 kB
        Erik Krogen
      6. HDFS-12131.004.patch
        13 kB
        Erik Krogen
      7. HDFS-12131.005.patch
        17 kB
        Erik Krogen
      8. HDFS-12131.006.patch
        17 kB
        Erik Krogen
      9. HDFS-12131-branch-2.006.patch
        17 kB
        Erik Krogen
      10. HDFS-12131-branch-2.7.006.patch
        15 kB
        Erik Krogen
      11. HDFS-12131-branch-2.8.006.patch
        15 kB
        Erik Krogen

        Activity

        Hide
        alloydwhitlock Adam Whitlock added a comment -
        Show
        alloydwhitlock Adam Whitlock added a comment - Konstantin Shvachko and Erik Krogen - Thank you!
        Hide
        xkrogen Erik Krogen added a comment -

        Thank you Konstantin!

        Show
        xkrogen Erik Krogen added a comment - Thank you Konstantin!
        Hide
        shv Konstantin Shvachko added a comment -

        Committed to branch-2.7. Thank you Erik.

        Show
        shv Konstantin Shvachko added a comment - Committed to branch-2.7. Thank you Erik.
        Hide
        shv Konstantin Shvachko added a comment -

        This is Jenkins run for branch-2.7 patch: https://builds.apache.org/job/PreCommit-HDFS-Build/21023/
        Rerun failed tests locally all passed, diffs are all empty.

        Show
        shv Konstantin Shvachko added a comment - This is Jenkins run for branch-2.7 patch: https://builds.apache.org/job/PreCommit-HDFS-Build/21023/ Rerun failed tests locally all passed, diffs are all empty.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 10m 38s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2.7 Compile Tests
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 7m 42s branch-2.7 passed
        +1 compile 5m 17s branch-2.7 passed with JDK v1.8.0_144
        +1 compile 6m 8s branch-2.7 passed with JDK v1.7.0_151
        +1 checkstyle 1m 21s branch-2.7 passed
        +1 mvnsite 1m 44s branch-2.7 passed
        -1 findbugs 1m 33s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings.
        +1 javadoc 1m 39s branch-2.7 passed with JDK v1.8.0_144
        +1 javadoc 2m 45s branch-2.7 passed with JDK v1.7.0_151
              Patch Compile Tests
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        +1 compile 6m 21s the patch passed with JDK v1.8.0_144
        +1 javac 6m 21s the patch passed
        +1 compile 6m 15s the patch passed with JDK v1.7.0_151
        +1 javac 6m 15s the patch passed
        +1 checkstyle 1m 21s the patch passed
        +1 mvnsite 1m 52s the patch passed
        -1 whitespace 0m 0s The patch has 125 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 5m 9s the patch passed
        +1 javadoc 1m 45s the patch passed with JDK v1.8.0_144
        +1 javadoc 2m 44s the patch passed with JDK v1.7.0_151
              Other Tests
        -1 unit 22m 56s hadoop-common in the patch failed with JDK v1.7.0_151.
        -1 unit 140m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_151.
        +1 asflicense 3m 10s The patch does not generate ASF License warnings.
        335m 31s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.ipc.TestDecayRpcScheduler
          hadoop.util.bloom.TestBloomFilters
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.namenode.ha.TestHAAppend
          hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation
        JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.conf.TestConfiguration



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885614/HDFS-12131-branch-2.7.006.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux f72cbbb5a63d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / c2350ec
        Default Java 1.7.0_151
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_151.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_151.txt
        JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21023/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21023/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 38s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2.7 Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 42s branch-2.7 passed +1 compile 5m 17s branch-2.7 passed with JDK v1.8.0_144 +1 compile 6m 8s branch-2.7 passed with JDK v1.7.0_151 +1 checkstyle 1m 21s branch-2.7 passed +1 mvnsite 1m 44s branch-2.7 passed -1 findbugs 1m 33s hadoop-common-project/hadoop-common in branch-2.7 has 3 extant Findbugs warnings. +1 javadoc 1m 39s branch-2.7 passed with JDK v1.8.0_144 +1 javadoc 2m 45s branch-2.7 passed with JDK v1.7.0_151       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 6m 21s the patch passed with JDK v1.8.0_144 +1 javac 6m 21s the patch passed +1 compile 6m 15s the patch passed with JDK v1.7.0_151 +1 javac 6m 15s the patch passed +1 checkstyle 1m 21s the patch passed +1 mvnsite 1m 52s the patch passed -1 whitespace 0m 0s The patch has 125 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 5m 9s the patch passed +1 javadoc 1m 45s the patch passed with JDK v1.8.0_144 +1 javadoc 2m 44s the patch passed with JDK v1.7.0_151       Other Tests -1 unit 22m 56s hadoop-common in the patch failed with JDK v1.7.0_151. -1 unit 140m 43s hadoop-hdfs in the patch failed with JDK v1.7.0_151. +1 asflicense 3m 10s The patch does not generate ASF License warnings. 335m 31s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.ipc.TestDecayRpcScheduler   hadoop.util.bloom.TestBloomFilters   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation JDK v1.8.0_144 Timed out junit tests org.apache.hadoop.conf.TestConfiguration Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885614/HDFS-12131-branch-2.7.006.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux f72cbbb5a63d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / c2350ec Default Java 1.7.0_151 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_151.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21023/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_151.txt JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21023/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21023/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment -

        Reopening to attach branch-2.7 patch as well. It required writing the host file manually instead of using HostsFileWriter since that was introduced in HDFS-9005 which we don't want to backport (it's large).

        Andrew Wang, if you want to commit since you did the other branches, let me know. Konstantin Shvachko also volunteered to do so otherwise.

        Show
        xkrogen Erik Krogen added a comment - Reopening to attach branch-2.7 patch as well. It required writing the host file manually instead of using HostsFileWriter since that was introduced in HDFS-9005 which we don't want to backport (it's large). Andrew Wang , if you want to commit since you did the other branches, let me know. Konstantin Shvachko also volunteered to do so otherwise.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12114 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12114/)
        HDFS-12131. Add some of the FSNamesystem JMX values as metrics. (wang: rev f4c6b00a9f48ae7667db4035b641769efc3bb7cf)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12114 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12114/ ) HDFS-12131 . Add some of the FSNamesystem JMX values as metrics. (wang: rev f4c6b00a9f48ae7667db4035b641769efc3bb7cf) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Wonderful! Thanks Erik, I've committed this to trunk, branch-2, branch-2.8.

        Show
        andrew.wang Andrew Wang added a comment - Wonderful! Thanks Erik, I've committed this to trunk, branch-2, branch-2.8.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 9m 33s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2.8 Compile Tests
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 8m 35s branch-2.8 passed
        +1 compile 6m 5s branch-2.8 passed with JDK v1.8.0_144
        +1 compile 6m 59s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 58s branch-2.8 passed
        +1 mvnsite 1m 41s branch-2.8 passed
        +1 findbugs 3m 12s branch-2.8 passed
        +1 javadoc 1m 16s branch-2.8 passed with JDK v1.8.0_144
        +1 javadoc 1m 44s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 5m 39s the patch passed with JDK v1.8.0_144
        +1 javac 5m 39s the patch passed
        +1 compile 6m 58s the patch passed with JDK v1.7.0_131
        +1 javac 6m 58s the patch passed
        +1 checkstyle 1m 1s the patch passed
        +1 mvnsite 1m 41s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 41s the patch passed
        +1 javadoc 1m 11s the patch passed with JDK v1.8.0_144
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_131
              Other Tests
        +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_131.
        -1 unit 46m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        174m 27s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.net.TestDNS
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.TestDistributedFileSystem
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880254/HDFS-12131-branch-2.8.006.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 555f33b3cd0d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 8943cf9
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20546/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20546/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20546/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 9m 33s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2.8 Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 35s branch-2.8 passed +1 compile 6m 5s branch-2.8 passed with JDK v1.8.0_144 +1 compile 6m 59s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 58s branch-2.8 passed +1 mvnsite 1m 41s branch-2.8 passed +1 findbugs 3m 12s branch-2.8 passed +1 javadoc 1m 16s branch-2.8 passed with JDK v1.8.0_144 +1 javadoc 1m 44s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 5m 39s the patch passed with JDK v1.8.0_144 +1 javac 5m 39s the patch passed +1 compile 6m 58s the patch passed with JDK v1.7.0_131 +1 javac 6m 58s the patch passed +1 checkstyle 1m 1s the patch passed +1 mvnsite 1m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 41s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_144 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 7m 44s hadoop-common in the patch passed with JDK v1.7.0_131. -1 unit 46m 20s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 174m 27s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.TestDistributedFileSystem JDK v1.7.0_131 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880254/HDFS-12131-branch-2.8.006.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 555f33b3cd0d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 8943cf9 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20546/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20546/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20546/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment -

        Great Andrew Wang, thanks! When putting together the branch-2 patch I noticed two small issues in my trunk patch - a few of the new additions to Metrics.md did not have correct formatting (missing a pipe at the end of the line) and I didn't close the FsVolumeReferences object that I obtained in TestNameNodeMetrics#testVolumeFailures(). Fixed those two in v006 patch, and attached corresponding branch-2 and branch-2.8 patches.

        Show
        xkrogen Erik Krogen added a comment - Great Andrew Wang , thanks! When putting together the branch-2 patch I noticed two small issues in my trunk patch - a few of the new additions to Metrics.md did not have correct formatting (missing a pipe at the end of the line) and I didn't close the FsVolumeReferences object that I obtained in TestNameNodeMetrics#testVolumeFailures() . Fixed those two in v006 patch, and attached corresponding branch-2 and branch-2.8 patches.
        Hide
        andrew.wang Andrew Wang added a comment -

        Looks like the patch doesn't apply cleanly to branch-2, mind providing one of those too? I haven't committed the trunk patch yet.

        Show
        andrew.wang Andrew Wang added a comment - Looks like the patch doesn't apply cleanly to branch-2, mind providing one of those too? I haven't committed the trunk patch yet.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM, thanks Erik! Agree with your comments, no need to do a lot of work for this test coverage. Will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM, thanks Erik! Agree with your comments, no need to do a lot of work for this test coverage. Will commit shortly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 13m 16s trunk passed
        +1 compile 13m 47s trunk passed
        +1 checkstyle 2m 0s trunk passed
        +1 mvnsite 2m 27s trunk passed
        -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 1m 37s trunk passed
              Patch Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 26s the patch passed
        +1 compile 10m 17s the patch passed
        +1 javac 10m 17s the patch passed
        +1 checkstyle 2m 3s the patch passed
        +1 mvnsite 2m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 25s the patch passed
        +1 javadoc 1m 38s the patch passed
              Other Tests
        -1 unit 8m 2s hadoop-common in the patch failed.
        -1 unit 65m 54s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        133m 55s



        Reason Tests
        Failed junit tests hadoop.security.TestRaceWhenRelogin
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879922/HDFS-12131.005.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux febbf73a4d27 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a11c230
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20525/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20525/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 13m 16s trunk passed +1 compile 13m 47s trunk passed +1 checkstyle 2m 0s trunk passed +1 mvnsite 2m 27s trunk passed -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 37s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 10m 17s the patch passed +1 javac 10m 17s the patch passed +1 checkstyle 2m 3s the patch passed +1 mvnsite 2m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 38s the patch passed       Other Tests -1 unit 8m 2s hadoop-common in the patch failed. -1 unit 65m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 133m 55s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879922/HDFS-12131.005.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux febbf73a4d27 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a11c230 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20525/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20525/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20525/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment - - edited

        Hey Andrew Wang, thanks for continuing to help work on this. I added tests for VolumeFailuresTotal, EstimatedCapacityLostTotal, and DecommissioningDataNodes in v005 patch.

        I did not write tests for NumInMaintenance(Live|Dead)DataNodes, NumEnteringMaintenanceDataNodes, and NumStaleStorages. The code to coerce the maintenance state transitions in TestMaintenanceState relies heavily on functions from AdminStatesBaseTest which I would rather not replicate just to test a metric output (when the underlying value is already being tested). I can't find existing test code showing how to coerce a nonzero NumStaleStorages and again would rather not spend too much effort trying to test just a metric value - I think time would be better spent actually adding a test for stale storages if one does not yet exist (I was unable to find one that I might be able to use as an example). If you have pointers let me know.

        Show
        xkrogen Erik Krogen added a comment - - edited Hey Andrew Wang , thanks for continuing to help work on this. I added tests for VolumeFailuresTotal , EstimatedCapacityLostTotal , and DecommissioningDataNodes in v005 patch. I did not write tests for NumInMaintenance(Live|Dead)DataNodes , NumEnteringMaintenanceDataNodes , and NumStaleStorages . The code to coerce the maintenance state transitions in TestMaintenanceState relies heavily on functions from AdminStatesBaseTest which I would rather not replicate just to test a metric output (when the underlying value is already being tested). I can't find existing test code showing how to coerce a nonzero NumStaleStorages and again would rather not spend too much effort trying to test just a metric value - I think time would be better spent actually adding a test for stale storages if one does not yet exist (I was unable to find one that I might be able to use as an example). If you have pointers let me know.
        Hide
        andrew.wang Andrew Wang added a comment -

        Sure, let's do the deprecate+removal in separate JIRAs for clarity.

        Some of the new metrics do not have unit test coverage. Could we add this?

        Show
        andrew.wang Andrew Wang added a comment - Sure, let's do the deprecate+removal in separate JIRAs for clarity. Some of the new metrics do not have unit test coverage. Could we add this?
        Hide
        xkrogen Erik Krogen added a comment -

        Wasn't able to replicate any of the unit test failures locally, even the ones that may seem related (e.g. TestHAMetrics). They seem flaky given that they succeeded in one Jenkins run and failed in the next.

        Show
        xkrogen Erik Krogen added a comment - Wasn't able to replicate any of the unit test failures locally, even the ones that may seem related (e.g. TestHAMetrics). They seem flaky given that they succeeded in one Jenkins run and failed in the next.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 30s Maven dependency ordering for branch
        +1 mvninstall 14m 54s trunk passed
        +1 compile 15m 30s trunk passed
        +1 checkstyle 2m 8s trunk passed
        +1 mvnsite 2m 43s trunk passed
        -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 1m 46s trunk passed
              Patch Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 39s the patch passed
        +1 compile 11m 42s the patch passed
        +1 javac 11m 42s the patch passed
        +1 checkstyle 2m 6s the patch passed
        +1 mvnsite 2m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 4m 1s the patch passed
        +1 javadoc 1m 44s the patch passed
              Other Tests
        +1 unit 9m 24s hadoop-common in the patch passed.
        -1 unit 92m 55s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 32s The patch does not generate ASF License warnings.
        170m 35s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.TestMissingBlocksAlert
          hadoop.hdfs.server.namenode.ha.TestHAMetrics
        Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879409/HDFS-12131.004.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux b333fc82e1e5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 746189a
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20475/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20475/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20475/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 1m 30s Maven dependency ordering for branch +1 mvninstall 14m 54s trunk passed +1 compile 15m 30s trunk passed +1 checkstyle 2m 8s trunk passed +1 mvnsite 2m 43s trunk passed -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 46s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 11m 42s the patch passed +1 javac 11m 42s the patch passed +1 checkstyle 2m 6s the patch passed +1 mvnsite 2m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 1s the patch passed +1 javadoc 1m 44s the patch passed       Other Tests +1 unit 9m 24s hadoop-common in the patch passed. -1 unit 92m 55s hadoop-hdfs in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 170m 35s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.namenode.ha.TestHAMetrics Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879409/HDFS-12131.004.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux b333fc82e1e5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 746189a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20475/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20475/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20475/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20475/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 13m 29s trunk passed
        +1 compile 15m 21s trunk passed
        +1 checkstyle 2m 7s trunk passed
        +1 mvnsite 2m 38s trunk passed
        -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 1m 41s trunk passed
              Patch Compile Tests
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 46s the patch passed
        +1 compile 11m 21s the patch passed
        +1 javac 11m 21s the patch passed
        +1 checkstyle 2m 10s the patch passed
        +1 mvnsite 2m 46s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 47s the patch passed
        +1 javadoc 1m 42s the patch passed
              Other Tests
        +1 unit 8m 30s hadoop-common in the patch passed.
        -1 unit 95m 49s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 44s The patch does not generate ASF License warnings.
        169m 16s



        Reason Tests
        Failed junit tests hadoop.tracing.TestTracing
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.TestSafeModeWithStripedFile
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879405/HDFS-12131.003.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 148c8498ac2a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 746189a
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20474/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20474/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20474/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20474/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 29s trunk passed +1 compile 15m 21s trunk passed +1 checkstyle 2m 7s trunk passed +1 mvnsite 2m 38s trunk passed -1 findbugs 1m 52s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 41s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 46s the patch passed +1 compile 11m 21s the patch passed +1 javac 11m 21s the patch passed +1 checkstyle 2m 10s the patch passed +1 mvnsite 2m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 47s the patch passed +1 javadoc 1m 42s the patch passed       Other Tests +1 unit 8m 30s hadoop-common in the patch passed. -1 unit 95m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 169m 16s Reason Tests Failed junit tests hadoop.tracing.TestTracing   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.TestSafeModeWithStripedFile Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879405/HDFS-12131.003.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 148c8498ac2a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 746189a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20474/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20474/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20474/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20474/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment -

        Okay, that seems fair. Should we have a separate JIRA for fixing up all of the remaining inconsistent ones in FSNamesystem to be able to more clearly link that to the eventual removal JIRA?

        Show
        xkrogen Erik Krogen added a comment - Okay, that seems fair. Should we have a separate JIRA for fixing up all of the remaining inconsistent ones in FSNamesystem to be able to more clearly link that to the eventual removal JIRA?
        Hide
        andrew.wang Andrew Wang added a comment -

        The prior example I have of deprecation+removal is HDFS-9024 and HDFS-5165, maybe that?

        Show
        andrew.wang Andrew Wang added a comment - The prior example I have of deprecation+removal is HDFS-9024 and HDFS-5165 , maybe that?
        Hide
        xkrogen Erik Krogen added a comment -

        Actually, I realized that the Javadoc on MaxObjects is wrong, it just represents the configuration for the maximum, not the current total. Not really a useful metric, will remove that one. v004 patch. Created HDFS-12219 to fix the javadoc.

        Show
        xkrogen Erik Krogen added a comment - Actually, I realized that the Javadoc on MaxObjects is wrong, it just represents the configuration for the maximum, not the current total. Not really a useful metric, will remove that one. v004 patch. Created HDFS-12219 to fix the javadoc.
        Hide
        xkrogen Erik Krogen added a comment -

        Ah, nice! Looks like you already got the reviews you needed there while I was trying to wrap my head around the historical discussion in HDFS-10999

        Cool, I'm with you on matching up these new ones rather than continuing a questionable practice. Do we have precedence for deprecating metrics? Given that there is already StaleDataNodes in FSNamesystem and NumStaleDataNodes in FSNamesystemState, I don't really like the idea of having a third NumStaleDataNodes in FSNamesystem. But if that's the only way for us to move towards uniformity here, I'm all for it.

        For now I've attached a v003 patch which adds in the Num prefix for the new metrics, and also adds in the other 6 metrics I mentioned in my first comment. I haven't touched StaleDataNodes yet.

        Show
        xkrogen Erik Krogen added a comment - Ah, nice! Looks like you already got the reviews you needed there while I was trying to wrap my head around the historical discussion in HDFS-10999 Cool, I'm with you on matching up these new ones rather than continuing a questionable practice. Do we have precedence for deprecating metrics? Given that there is already StaleDataNodes in FSNamesystem and NumStaleDataNodes in FSNamesystemState, I don't really like the idea of having a third NumStaleDataNodes in FSNamesystem. But if that's the only way for us to move towards uniformity here, I'm all for it. For now I've attached a v003 patch which adds in the Num prefix for the new metrics, and also adds in the other 6 metrics I mentioned in my first comment. I haven't touched StaleDataNodes yet.
        Hide
        andrew.wang Andrew Wang added a comment -

        I've got a patch out already at HDFS-12206 to fix the replicated/ecblock naming, noticed that while looking at this earlier If you want to review, that'd be great.

        My vote is to set a good example for the future and keep the names the same. Optionally, we can add new metrics with matched names and deprecate the mismatched ones. We could do that for StaleDataNodes here, and leave the rest for another patch.

        Show
        andrew.wang Andrew Wang added a comment - I've got a patch out already at HDFS-12206 to fix the replicated/ecblock naming, noticed that while looking at this earlier If you want to review, that'd be great. My vote is to set a good example for the future and keep the names the same. Optionally, we can add new metrics with matched names and deprecate the mismatched ones. We could do that for StaleDataNodes here, and leave the rest for another patch.
        Hide
        xkrogen Erik Krogen added a comment -

        Hey Andrew Wang, thanks for looking. Here I followed the example of StaleDataNodes since that was the most closely related one, but I agree with you that having the metric name match the MBean name makes more sense. There are a number of metrics in FSNamesystem that don't match the MBean though, even besides the Num prefix: LockQueueLength v FsLockQueueLength, all of the ReplicatedBlocksMBean ones, and all of the ECBlockGroupsStatsMBean ones.

        I don't really like that StaleDataNodes will be an outlier compared to the other XxxDataNodes metrics if I put the Num prefix in this patch, but otherwise I'm in support of it. If you'd prefer I put back in the Num prefix I'm fine with it.

        Show
        xkrogen Erik Krogen added a comment - Hey Andrew Wang , thanks for looking. Here I followed the example of StaleDataNodes since that was the most closely related one, but I agree with you that having the metric name match the MBean name makes more sense. There are a number of metrics in FSNamesystem that don't match the MBean though, even besides the Num prefix: LockQueueLength v FsLockQueueLength, all of the ReplicatedBlocksMBean ones, and all of the ECBlockGroupsStatsMBean ones. I don't really like that StaleDataNodes will be an outlier compared to the other XxxDataNodes metrics if I put the Num prefix in this patch, but otherwise I'm in support of it. If you'd prefer I put back in the Num prefix I'm fine with it.
        Hide
        andrew.wang Andrew Wang added a comment -

        I don't see a downside to exposing these as metrics, thanks for doing this Erik!

        One question about the patch, I notice that some of the other metrics in FSNamesystem keep the "Num" prefix:
        NumActiveClients, NumFilesUnderConstruction, NumEncryptionZones. Some do not: Snapshots, SnapshottableDirectories.

        Do you have an opinion on which is better? I think it'd be simpler if the metric name lined up with the MBean name. We could consider cleaning this up for the future too, via deprecation and later removal.

        Show
        andrew.wang Andrew Wang added a comment - I don't see a downside to exposing these as metrics, thanks for doing this Erik! One question about the patch, I notice that some of the other metrics in FSNamesystem keep the "Num" prefix: NumActiveClients, NumFilesUnderConstruction, NumEncryptionZones. Some do not: Snapshots, SnapshottableDirectories. Do you have an opinion on which is better? I think it'd be simpler if the metric name lined up with the MBean name. We could consider cleaning this up for the future too, via deprecation and later removal.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 14m 48s trunk passed
        +1 compile 17m 6s trunk passed
        +1 checkstyle 2m 13s trunk passed
        +1 mvnsite 2m 46s trunk passed
        -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 1m 35s trunk passed
              Patch Compile Tests
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 26s the patch passed
        +1 compile 10m 11s the patch passed
        +1 javac 10m 11s the patch passed
        +1 checkstyle 1m 56s the patch passed
        +1 mvnsite 2m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 28s the patch passed
        +1 javadoc 1m 38s the patch passed
              Other Tests
        -1 unit 7m 17s hadoop-common in the patch failed.
        -1 unit 71m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 36s The patch does not generate ASF License warnings.
        144m 1s



        Reason Tests
        Failed junit tests hadoop.net.TestDNS
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878206/HDFS-12131.002.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
        uname Linux 5d01e7da72b8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c8df366
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20356/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20356/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 14m 48s trunk passed +1 compile 17m 6s trunk passed +1 checkstyle 2m 13s trunk passed +1 mvnsite 2m 46s trunk passed -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 35s trunk passed       Patch Compile Tests 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 10m 11s the patch passed +1 javac 10m 11s the patch passed +1 checkstyle 1m 56s the patch passed +1 mvnsite 2m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 38s the patch passed       Other Tests -1 unit 7m 17s hadoop-common in the patch failed. -1 unit 71m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 144m 1s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878206/HDFS-12131.002.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 5d01e7da72b8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c8df366 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20356/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20356/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20356/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        +1 non-binding pending jenkins
        Thanks Erik Krogen!

        Show
        shahrs87 Rushabh S Shah added a comment - +1 non-binding pending jenkins Thanks Erik Krogen !
        Hide
        xkrogen Erik Krogen added a comment -

        Reattaching v002 patch since Jenkins didn't run.

        Show
        xkrogen Erik Krogen added a comment - Reattaching v002 patch since Jenkins didn't run.
        Hide
        xkrogen Erik Krogen added a comment -

        Attaching v002 patch with updated Metrics.md. Thanks for the pointer Rushabh S Shah!

        Show
        xkrogen Erik Krogen added a comment - Attaching v002 patch with updated Metrics.md . Thanks for the pointer Rushabh S Shah !
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Thanks Erik Krogen for revising the patch.
        Overall the patch looks good.
        Just one thing:
        Update the ./hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md file to update the documentation.
        Add the newly added metrics in FSNamesystem context.

        Show
        shahrs87 Rushabh S Shah added a comment - Thanks Erik Krogen for revising the patch. Overall the patch looks good. Just one thing: Update the ./hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md file to update the documentation. Add the newly added metrics in FSNamesystem context.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 52s trunk passed
        +1 compile 1m 0s trunk passed
        +1 checkstyle 0m 41s trunk passed
        +1 mvnsite 1m 4s trunk passed
        -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 45s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 59s the patch passed
        +1 compile 0m 53s the patch passed
        +1 javac 0m 53s the patch passed
        +1 checkstyle 0m 39s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 3s the patch passed
        +1 javadoc 0m 42s the patch passed
              Other Tests
        -1 unit 69m 36s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        98m 3s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877697/HDFS-12131.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b7f4e41adde5 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5b00792
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20319/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20319/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20319/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20319/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 52s trunk passed +1 compile 1m 0s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 4s trunk passed -1 findbugs 1m 45s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 45s trunk passed       Patch Compile Tests +1 mvninstall 0m 59s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 39s the patch passed +1 mvnsite 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 42s the patch passed       Other Tests -1 unit 69m 36s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 98m 3s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877697/HDFS-12131.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b7f4e41adde5 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b00792 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20319/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20319/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20319/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20319/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment -

        Ah, thank you for the pointer Rushabh S Shah! I was trying to find where to put unit tests but couldn't find any in TestFSNamesystem/TestFSNamesystemMBean. Attached v001 patch.

        Show
        xkrogen Erik Krogen added a comment - Ah, thank you for the pointer Rushabh S Shah ! I was trying to find where to put unit tests but couldn't find any in TestFSNamesystem / TestFSNamesystemMBean . Attached v001 patch.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Thanks Erik Krogen for the patch.
        Adding a test case in TestNameNodeMetrics.java would be good.

        Show
        shahrs87 Rushabh S Shah added a comment - Thanks Erik Krogen for the patch. Adding a test case in TestNameNodeMetrics.java would be good.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 14m 20s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 54s trunk passed
        -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 45s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 38s the patch passed
              Other Tests
        -1 unit 89m 19s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        115m 55s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12131
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877099/HDFS-12131.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 276ea8290fb8 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b61ab85
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20259/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20259/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20259/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 14m 20s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 54s trunk passed -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 45s trunk passed       Patch Compile Tests +1 mvninstall 0m 53s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 38s the patch passed       Other Tests -1 unit 89m 19s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 115m 55s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12131 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877099/HDFS-12131.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 276ea8290fb8 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b61ab85 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20259/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20259/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20259/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20259/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xkrogen Erik Krogen added a comment - - edited

        Attaching patch which simply annotates these JMX Bean methods with @Metric annotations.

        New JMX output from a MiniDFSCluster (note that these are in the JMX portion corresponding to the metrics rather than the FSNamesystemMBean):

            "name" : "Hadoop:service=NameNode,name=FSNamesystem",
            "modelerType" : "FSNamesystem",
        ...
            "LiveDataNodes" : 1,
            "DeadDataNodes" : 0,
            "DecomLiveDataNodes" : 0,
            "DecomDeadDataNodes" : 0,
            "DecommissioningDataNodes" : 0,
            "StaleDataNodes" : 0,
            "StaleStorages" : 0,
        

        The six values listed in the description are the ones we are particularly interested in, but there are a few others which seem like they may be valuable as metrics as well:

        • MaxObjects
        • VolumeFailuresTotal
        • EstimatedCapacityLostTotal
        • NumInMaintenanceLiveDataNodes
        • NumInMaintenanceDeadDataNodes
        • NumEnteringMaintenanceDataNodes

        I am wondering if these should be added as metrics as well? Is there a downside? cc Andrew Wang

        Show
        xkrogen Erik Krogen added a comment - - edited Attaching patch which simply annotates these JMX Bean methods with @Metric annotations. New JMX output from a MiniDFSCluster (note that these are in the JMX portion corresponding to the metrics rather than the FSNamesystemMBean): "name" : "Hadoop:service=NameNode,name=FSNamesystem" , "modelerType" : "FSNamesystem" , ... "LiveDataNodes" : 1, "DeadDataNodes" : 0, "DecomLiveDataNodes" : 0, "DecomDeadDataNodes" : 0, "DecommissioningDataNodes" : 0, "StaleDataNodes" : 0, "StaleStorages" : 0, The six values listed in the description are the ones we are particularly interested in, but there are a few others which seem like they may be valuable as metrics as well: MaxObjects VolumeFailuresTotal EstimatedCapacityLostTotal NumInMaintenanceLiveDataNodes NumInMaintenanceDeadDataNodes NumEnteringMaintenanceDataNodes I am wondering if these should be added as metrics as well? Is there a downside? cc Andrew Wang

          People

          • Assignee:
            xkrogen Erik Krogen
            Reporter:
            xkrogen Erik Krogen
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development