Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha2, 2.7.5
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The nntop output is already exposed via JMX with HDFS-6982.

      However external metrics systems do not get this data. It would be valuable to track this as a timeseries as well.

      1. HDFS-10984.v4.patch
        9 kB
        Siddharth Wagle
      2. HDFS-10984.v3.patch
        9 kB
        Siddharth Wagle
      3. HDFS-10984.v2.patch
        8 kB
        Siddharth Wagle
      4. HDFS-10984.v1.patch
        8 kB
        Siddharth Wagle
      5. HDFS-10984.patch
        4 kB
        Siddharth Wagle

        Issue Links

          Activity

          Hide
          swagle Siddharth Wagle added a comment - - edited

          Sample flattened output of the window metrics emitted to Ambari Metrics System:

          dfs.NNTopUserOpCounts.windowMs=60000.op=listStatus.user=mapred.count

          Show
          swagle Siddharth Wagle added a comment - - edited Sample flattened output of the window metrics emitted to Ambari Metrics System: dfs.NNTopUserOpCounts.windowMs=60000.op=listStatus.user=mapred.count
          Hide
          swagle Siddharth Wagle added a comment -

          Set fix version to 2.7.3

          Show
          swagle Siddharth Wagle added a comment - Set fix version to 2.7.3
          Hide
          swagle Siddharth Wagle added a comment -

          Minor correction.

          Show
          swagle Siddharth Wagle added a comment - Minor correction.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 16s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 40s the patch passed
          +1 javac 0m 40s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 90m 22s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          109m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.TestRollingUpgrade
          Timed out junit tests org.apache.hadoop.tools.TestJMXGet



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832240/HDFS-10984.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 71153b448b4a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6a38d11
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17066/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17066/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 40s the patch passed +1 javac 0m 40s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 90m 22s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 109m 27s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.TestRollingUpgrade Timed out junit tests org.apache.hadoop.tools.TestJMXGet Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832240/HDFS-10984.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 71153b448b4a 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a38d11 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17066/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17066/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17066/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 9m 33s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 4s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 1m 3s the patch passed
          +1 compile 0m 59s the patch passed
          +1 javac 0m 59s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185)
          +1 mvnsite 1m 8s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 78m 48s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          101m 59s



          Reason Tests
          Timed out junit tests org.apache.hadoop.tools.TestJMXGet



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832241/HDFS-10984.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 747e7c60cda0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6a38d11
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17067/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17067/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 33s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 4s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 0m 59s the patch passed +1 javac 0m 59s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 185 unchanged - 0 fixed = 187 total (was 185) +1 mvnsite 1m 8s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 78m 48s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 101m 59s Reason Tests Timed out junit tests org.apache.hadoop.tools.TestJMXGet Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832241/HDFS-10984.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 747e7c60cda0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6a38d11 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17067/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17067/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17067/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          swagle Siddharth Wagle added a comment -

          Unit test changes pending.

          Show
          swagle Siddharth Wagle added a comment - Unit test changes pending.
          Hide
          swagle Siddharth Wagle added a comment -

          Added unit tests for topMetrics changes. Additionally a timeout on JMXGet().

          Show
          swagle Siddharth Wagle added a comment - Added unit tests for topMetrics changes. Additionally a timeout on JMXGet().
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 21s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 52s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 52s the patch passed
          +1 javac 0m 52s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 206 unchanged - 0 fixed = 213 total (was 206)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 39s the patch passed
          -1 unit 72m 21s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          92m 38s



          Reason Tests
          Failed junit tests hadoop.tools.TestJMXGet



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832308/HDFS-10984.v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 493a4915644d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4d10621
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17070/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17070/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 21s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 206 unchanged - 0 fixed = 213 total (was 206) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 72m 21s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 92m 38s Reason Tests Failed junit tests hadoop.tools.TestJMXGet Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832308/HDFS-10984.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 493a4915644d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4d10621 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17070/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17070/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17070/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          swagle Siddharth Wagle added a comment -

          Fixed JMXGet issue, verified that it works locally with several runs.

          Show
          swagle Siddharth Wagle added a comment - Fixed JMXGet issue, verified that it works locally with several runs.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Thanks Siddharth Wagle for reporting the issue and posting the patch. The latest patch v2 with the JMXGet issue fixed looks good to me overall.
          Here are a few minor issues:

          1. In TopMetrics#getMetrics(), should we check if top metrics is enabled or not like we do in FSNamesystem#getTopUserOpCounts. Or have a unit test verify that the metrics source works as expected when top metrics is not enabled.

          2. I notice that the patch added a metric record per window. So there would be multiple record per getMetrics() call for each window. Can we elaborate this in the description of the comments?

          3. Checkstyle issue from Jenkins.

          4. You may also add context info into this ticket.
          The original HDFS-6982 has a Metrics2 source implemented but was removed as part of HDFS-7426 "Change nntop JMX format to be a JSON blob".

          Show
          xyao Xiaoyu Yao added a comment - - edited Thanks Siddharth Wagle for reporting the issue and posting the patch. The latest patch v2 with the JMXGet issue fixed looks good to me overall. Here are a few minor issues: 1. In TopMetrics#getMetrics(), should we check if top metrics is enabled or not like we do in FSNamesystem#getTopUserOpCounts. Or have a unit test verify that the metrics source works as expected when top metrics is not enabled. 2. I notice that the patch added a metric record per window. So there would be multiple record per getMetrics() call for each window. Can we elaborate this in the description of the comments? 3. Checkstyle issue from Jenkins. 4. You may also add context info into this ticket. The original HDFS-6982 has a Metrics2 source implemented but was removed as part of HDFS-7426 "Change nntop JMX format to be a JSON blob".
          Hide
          swagle Siddharth Wagle added a comment -

          Thanks Xiaoyu Yao for the review comments, I will work on incorporating them into the patch.

          Confused about point 4. I have linked the two tickets, did you mean a Jira comment or something for the javadoc as a class level commentary?

          Show
          swagle Siddharth Wagle added a comment - Thanks Xiaoyu Yao for the review comments, I will work on incorporating them into the patch. Confused about point 4. I have linked the two tickets, did you mean a Jira comment or something for the javadoc as a class level commentary?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 33s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 185 unchanged - 0 fixed = 193 total (was 185)
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 63m 18s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          84m 15s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.tools.TestDFSAdmin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832522/HDFS-10984.v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 358780967afa 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3441c74
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17086/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17086/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17086/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17086/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 33s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 50s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 185 unchanged - 0 fixed = 193 total (was 185) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 63m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 84m 15s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.tools.TestDFSAdmin Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832522/HDFS-10984.v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 358780967afa 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3441c74 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17086/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17086/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17086/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17086/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Confused about point 4. I have linked the two tickets, did you mean a Jira comment or something for the javadoc as a class level commentary?

          I mean the description of the JIRA "The nntop output is already exposed via JMX with HDFS-6982."

          Show
          xyao Xiaoyu Yao added a comment - Confused about point 4. I have linked the two tickets, did you mean a Jira comment or something for the javadoc as a class level commentary? I mean the description of the JIRA "The nntop output is already exposed via JMX with HDFS-6982 ."
          Hide
          swagle Siddharth Wagle added a comment -

          Cancelling to incorporate review comments.

          Show
          swagle Siddharth Wagle added a comment - Cancelling to incorporate review comments.
          Hide
          swagle Siddharth Wagle added a comment -

          Addressed review comments from Xiaoyu Yao.

          Show
          swagle Siddharth Wagle added a comment - Addressed review comments from Xiaoyu Yao .
          Hide
          swagle Siddharth Wagle added a comment -

          Re-submitting patch for review + tests.

          Show
          swagle Siddharth Wagle added a comment - Re-submitting patch for review + tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 24s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 185 unchanged - 0 fixed = 188 total (was 185)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 68m 53s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          88m 9s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832544/HDFS-10984.v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 349de580663e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c874fa9
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17089/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17089/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17089/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17089/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 24s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 185 unchanged - 0 fixed = 188 total (was 185) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 68m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 88m 9s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832544/HDFS-10984.v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 349de580663e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c874fa9 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17089/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17089/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17089/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17089/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Siddharth Wagle for updating the patch. Patch v3 looks good to me. +1 after the checkstyle issue is addressed.

          The failed tests from last Jenkins run are not related to this change. I've tested them locally and non of them repro.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Siddharth Wagle for updating the patch. Patch v3 looks good to me. +1 after the checkstyle issue is addressed. The failed tests from last Jenkins run are not related to this change. I've tested them locally and non of them repro.
          Hide
          swagle Siddharth Wagle added a comment -

          Cancelling to fix checkStyle.

          Show
          swagle Siddharth Wagle added a comment - Cancelling to fix checkStyle.
          Hide
          swagle Siddharth Wagle added a comment -

          Thanks Xiaoyu Yao. I ran these tests manually and all three worked fine.

          Show
          swagle Siddharth Wagle added a comment - Thanks Xiaoyu Yao . I ran these tests manually and all three worked fine.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 27s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 36s the patch passed
          +1 unit 62m 23s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          83m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10984
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832716/HDFS-10984.v4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1db851ff3381 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 809cfd2
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17104/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17104/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 27s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 62m 23s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 83m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10984 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832716/HDFS-10984.v4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1db851ff3381 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 809cfd2 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17104/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17104/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          +1 for the v4 patch. I will commit it shortly.

          Show
          xyao Xiaoyu Yao added a comment - +1 for the v4 patch. I will commit it shortly.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Siddharth Wagle for the contribution. I've commit the patch to trunk, branch-2 and branch-2.8.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Siddharth Wagle for the contribution. I've commit the patch to trunk, branch-2 and branch-2.8.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10591/)
          HDFS-10984. Expose nntop output as metrics. Contributed by Siddharth (xyao: rev 61f0490a73085bbaf6639d9234277e59dc1145db)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/metrics/TopMetrics.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestTopMetrics.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10591/ ) HDFS-10984 . Expose nntop output as metrics. Contributed by Siddharth (xyao: rev 61f0490a73085bbaf6639d9234277e59dc1145db) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/top/metrics/TopMetrics.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestTopMetrics.java
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Siddharth Wagle and Xiaoyu Yao. I backported this to branch-2.7 as well.

          Show
          zhz Zhe Zhang added a comment - Thanks Siddharth Wagle and Xiaoyu Yao . I backported this to branch-2.7 as well.

            People

            • Assignee:
              swagle Siddharth Wagle
              Reporter:
              swagle Siddharth Wagle
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development