Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9882

Add heartbeatsTotal in Datanode metrics

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Heartbeat latency only reflects the time spent on generating reports and sending reports to NN. When heartbeats are delayed due to processing commands, this latency does not help investigation. I would like to propose to add another metric counter to show the total time.

        Issue Links

          Activity

          Hide
          elgoiri Inigo Goiri added a comment -

          I think the second approach makes more sense as other people may have already built some dependencies on the semantic of the current heartbeats.
          The current approach in DatanodeMetrics has:

          @Metric MutableRate heartbeats;
          

          I would add:

          @Metric MutableRate heartbeatsTotal;
          
          Show
          elgoiri Inigo Goiri added a comment - I think the second approach makes more sense as other people may have already built some dependencies on the semantic of the current heartbeats . The current approach in DatanodeMetrics has: @Metric MutableRate heartbeats; I would add: @Metric MutableRate heartbeatsTotal;
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 47s trunk passed
          +1 compile 0m 50s trunk passed with JDK v1.8.0_72
          +1 compile 0m 46s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 11s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95
          -1 mvninstall 0m 29s hadoop-hdfs in the patch failed.
          -1 compile 0m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 javac 0m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 compile 0m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 javac 0m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 83 unchanged - 0 fixed = 85 total (was 83)
          -1 mvnsite 0m 33s hadoop-hdfs in the patch failed.
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 0m 20s hadoop-hdfs in the patch failed.
          +1 javadoc 1m 6s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95
          -1 unit 0m 24s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 0m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 20s Patch does not generate ASF License warnings.
          25m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790819/0001-Add-heartbeatsTotal-metric.patch
          JIRA Issue HDFS-9882
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux df5ab6dd5b00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          compile https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14690/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14690/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 47s trunk passed +1 compile 0m 50s trunk passed with JDK v1.8.0_72 +1 compile 0m 46s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 11s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95 -1 mvninstall 0m 29s hadoop-hdfs in the patch failed. -1 compile 0m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 javac 0m 27s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 compile 0m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 javac 0m 30s hadoop-hdfs in the patch failed with JDK v1.7.0_95. -1 checkstyle 0m 21s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 83 unchanged - 0 fixed = 85 total (was 83) -1 mvnsite 0m 33s hadoop-hdfs in the patch failed. +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 0m 20s hadoop-hdfs in the patch failed. +1 javadoc 1m 6s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95 -1 unit 0m 24s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 0m 28s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 25m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790819/0001-Add-heartbeatsTotal-metric.patch JIRA Issue HDFS-9882 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux df5ab6dd5b00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14690/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14690/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14690/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 0m 43s trunk passed with JDK v1.8.0_72
          +1 compile 0m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 23s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 4s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72
          +1 javadoc 1m 54s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 50s the patch passed
          +1 compile 0m 41s the patch passed with JDK v1.8.0_72
          +1 javac 0m 41s the patch passed
          +1 compile 0m 42s the patch passed with JDK v1.7.0_95
          +1 javac 0m 42s the patch passed
          -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 83 unchanged - 0 fixed = 85 total (was 83)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 15s the patch passed
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_72
          +1 javadoc 1m 51s the patch passed with JDK v1.7.0_95
          -1 unit 59m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          -1 unit 58m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          144m 54s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
            hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791021/0001-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch
          JIRA Issue HDFS-9882
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e39f2cbde365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67880cc
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14696/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14696/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 7s trunk passed +1 compile 0m 43s trunk passed with JDK v1.8.0_72 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 4s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 54s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 50s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_72 +1 javac 0m 41s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_95 +1 javac 0m 42s the patch passed -1 checkstyle 0m 20s hadoop-hdfs-project/hadoop-hdfs: patch generated 2 new + 83 unchanged - 0 fixed = 85 total (was 83) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 15s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_95 -1 unit 59m 47s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 58m 21s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 144m 54s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock   hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791021/0001-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch JIRA Issue HDFS-9882 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e39f2cbde365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14696/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14696/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14696/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 51s trunk passed
          +1 compile 0m 48s trunk passed with JDK v1.8.0_72
          +1 compile 0m 40s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 22s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_72
          +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 58s the patch passed with JDK v1.8.0_72
          +1 javac 0m 58s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_95
          +1 javac 0m 46s the patch passed
          -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 84 unchanged - 0 fixed = 85 total (was 84)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 17s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 2s the patch passed
          +1 javadoc 1m 21s the patch passed with JDK v1.8.0_72
          +1 javadoc 2m 36s the patch passed with JDK v1.7.0_95
          -1 unit 71m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
          +1 unit 59m 35s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          161m 11s



          Reason Tests
          JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.TestRollingUpgradeRollback
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput
            hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791083/0002-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch
          JIRA Issue HDFS-9882
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0661aa3407b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 700b0e4
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14700/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14700/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 51s trunk passed +1 compile 0m 48s trunk passed with JDK v1.8.0_72 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 1m 20s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 52s the patch passed +1 compile 0m 58s the patch passed with JDK v1.8.0_72 +1 javac 0m 58s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_95 +1 javac 0m 46s the patch passed -1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: patch generated 1 new + 84 unchanged - 0 fixed = 85 total (was 84) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 2s the patch passed +1 javadoc 1m 21s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 36s the patch passed with JDK v1.7.0_95 -1 unit 71m 15s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 59m 35s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 161m 11s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.TestFileAppend   hadoop.hdfs.TestRollingUpgradeRollback   hadoop.hdfs.TestErasureCodeBenchmarkThroughput   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791083/0002-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch JIRA Issue HDFS-9882 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0661aa3407b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 700b0e4 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14700/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14700/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Inigo Goiri added a comment -

          I think we need to add a description to Metrics.md.
          Other than that, I think the patch is good (I don't see any related unit tests for this and the left checkstyle would break the style of the class).

          Andrew Wang, Arpit Agarwal, do you guys think this is a useful addition?
          We found that the heartbeats were reporting as running smoothly but the block report processing was actually getting stuck because of the disk and delaying the heartbeats which wasn't easy to monitor. Actually, we are planning to open a separate JIRA to move some of the disk related checks to a separate thread.

          Show
          elgoiri Inigo Goiri added a comment - I think we need to add a description to Metrics.md. Other than that, I think the patch is good (I don't see any related unit tests for this and the left checkstyle would break the style of the class). Andrew Wang , Arpit Agarwal , do you guys think this is a useful addition? We found that the heartbeats were reporting as running smoothly but the block report processing was actually getting stuck because of the disk and delaying the heartbeats which wasn't easy to monitor. Actually, we are planning to open a separate JIRA to move some of the disk related checks to a separate thread.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Hi Inigo Goiri,

          heartbeats were reporting as running smoothly but the block report processing was actually getting stuck because of the disk and delaying the heartbeats which wasn't easy to monitor

          Do you mean processing commands from the NN was slow because of disk operations? Did you figure out which disk operations? IIRC we schedule async disk deletions to avoid this exact problem. Thanks.

          Show
          arpitagarwal Arpit Agarwal added a comment - Hi Inigo Goiri , heartbeats were reporting as running smoothly but the block report processing was actually getting stuck because of the disk and delaying the heartbeats which wasn't easy to monitor Do you mean processing commands from the NN was slow because of disk operations? Did you figure out which disk operations? IIRC we schedule async disk deletions to avoid this exact problem. Thanks.
          Hide
          hualiu Hua Liu added a comment -

          Hi Arpit Agarwal

          When a data node needs to transfer a block, it validates the block in the heartbeat thread invoking the checkBlock method of FsDatasetImpl, where it checks whether the block exists and gets the block length. If the block is valid, it then spins off a thread to do the actual block transfer. During heavy disk IO that happened once in our environment, we found the heartbeat thread hang on "replicaInfo.getBlockFile().exists()" for more than 10 minutes.

          Show
          hualiu Hua Liu added a comment - Hi Arpit Agarwal When a data node needs to transfer a block, it validates the block in the heartbeat thread invoking the checkBlock method of FsDatasetImpl, where it checks whether the block exists and gets the block length. If the block is valid, it then spins off a thread to do the actual block transfer. During heavy disk IO that happened once in our environment, we found the heartbeat thread hang on "replicaInfo.getBlockFile().exists()" for more than 10 minutes.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks that makes sense. This is a good find.

          Do you think it's a better idea to fix heartbeat handling to remove expensive operations?

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks that makes sense. This is a good find. Do you think it's a better idea to fix heartbeat handling to remove expensive operations?
          Hide
          elgoiri Inigo Goiri added a comment -

          Just for the record, this happened in Windows where the Hadoop code might not be that optimized.

          Not sure if we can remove those operations; it might be a little too deep of a change.
          For now, our internal solution has been to make these operations into a different thread and make checks from the heartbeat one.

          Show
          elgoiri Inigo Goiri added a comment - Just for the record, this happened in Windows where the Hadoop code might not be that optimized. Not sure if we can remove those operations; it might be a little too deep of a change. For now, our internal solution has been to make these operations into a different thread and make checks from the heartbeat one.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Disk operations can be slow on any platform if the disk is loaded or bad. So I think it is a good idea to move those operations out of the heartbeat processing path which is perf-sensitive. Would you consider filing a separate Jira to fix the checkBlock issue described by Hua Liu?

          Meanwhile we can also add this new metric. Can you rename it to something like HeartbeatTotalTime and describe it in Metrics.md?

          Show
          arpitagarwal Arpit Agarwal added a comment - Disk operations can be slow on any platform if the disk is loaded or bad. So I think it is a good idea to move those operations out of the heartbeat processing path which is perf-sensitive. Would you consider filing a separate Jira to fix the checkBlock issue described by Hua Liu ? Meanwhile we can also add this new metric. Can you rename it to something like HeartbeatTotalTime and describe it in Metrics.md?
          Hide
          elgoiri Inigo Goiri added a comment -

          I created HDFS-9910 to make the disk operations in the heartbeat asynchronous.

          Show
          elgoiri Inigo Goiri added a comment - I created HDFS-9910 to make the disk operations in the heartbeat asynchronous.
          Hide
          elgoiri Inigo Goiri added a comment -

          I missed HDFS-9901 which already does this, I marked it as duplicate.

          Show
          elgoiri Inigo Goiri added a comment - I missed HDFS-9901 which already does this, I marked it as duplicate.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 53s Maven dependency ordering for branch
          +1 mvninstall 9m 21s trunk passed
          +1 compile 11m 2s trunk passed with JDK v1.8.0_74
          +1 compile 7m 21s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 1m 53s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 3m 33s trunk passed
          +1 javadoc 2m 15s trunk passed with JDK v1.8.0_74
          +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 50s Maven dependency ordering for patch
          +1 mvninstall 1m 35s the patch passed
          +1 compile 8m 17s the patch passed with JDK v1.8.0_74
          +1 javac 8m 17s the patch passed
          +1 compile 7m 19s the patch passed with JDK v1.7.0_95
          +1 javac 7m 19s the patch passed
          -1 checkstyle 1m 7s root: patch generated 1 new + 82 unchanged - 0 fixed = 83 total (was 82)
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 3s the patch passed
          +1 javadoc 2m 18s the patch passed with JDK v1.8.0_74
          +1 javadoc 3m 3s the patch passed with JDK v1.7.0_95
          +1 unit 9m 52s hadoop-common in the patch passed with JDK v1.8.0_74.
          -1 unit 87m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 11m 7s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 82m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          265m 21s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
            hadoop.hdfs.server.namenode.ha.TestEditLogTailer
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestLeaseRecovery
            hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles
            hadoop.hdfs.server.datanode.TestDirectoryScanner
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791690/0003-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch
          JIRA Issue HDFS-9882
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 729b5baed4bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19ee185
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14729/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14729/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 9m 21s trunk passed +1 compile 11m 2s trunk passed with JDK v1.8.0_74 +1 compile 7m 21s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 2m 15s trunk passed with JDK v1.8.0_74 +1 javadoc 2m 59s trunk passed with JDK v1.7.0_95 0 mvndep 0m 50s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 8m 17s the patch passed with JDK v1.8.0_74 +1 javac 8m 17s the patch passed +1 compile 7m 19s the patch passed with JDK v1.7.0_95 +1 javac 7m 19s the patch passed -1 checkstyle 1m 7s root: patch generated 1 new + 82 unchanged - 0 fixed = 83 total (was 82) +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 3s the patch passed +1 javadoc 2m 18s the patch passed with JDK v1.8.0_74 +1 javadoc 3m 3s the patch passed with JDK v1.7.0_95 +1 unit 9m 52s hadoop-common in the patch passed with JDK v1.8.0_74. -1 unit 87m 37s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 11m 7s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 82m 58s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 265m 21s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.hdfs.server.namenode.ha.TestEditLogTailer   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestLeaseRecovery   hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791690/0003-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch JIRA Issue HDFS-9882 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 729b5baed4bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19ee185 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14729/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14729/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hualiu Hua Liu added a comment -

          Hi Arpit Agarwal

          Since NumOps and AvgTime are appended to the metric name, HeartbeatTotalTImeAvgTime would look verbose and HeartbeatTotalTimeNumOps would appear confusing. We think heartbeatsTotal may be a good alternative.

          And we described this new metric in metrics.md. Please take a look at it and submit if you see fit.

          Thanks,
          Hua

          Show
          hualiu Hua Liu added a comment - Hi Arpit Agarwal Since NumOps and AvgTime are appended to the metric name, HeartbeatTotalTImeAvgTime would look verbose and HeartbeatTotalTimeNumOps would appear confusing. We think heartbeatsTotal may be a good alternative. And we described this new metric in metrics.md. Please take a look at it and submit if you see fit. Thanks, Hua
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          We think heartbeatsTotal may be a good alternative.

          Makes sense. Do you want to change the function name addHeartbeatTotalTime to be consistent with the metric name addHeartbeatTotal? The v3 patch looks fine otherwise.

          Show
          arpitagarwal Arpit Agarwal added a comment - We think heartbeatsTotal may be a good alternative. Makes sense. Do you want to change the function name addHeartbeatTotalTime to be consistent with the metric name addHeartbeatTotal ? The v3 patch looks fine otherwise.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          +1 pending jenkins for the v4 patch.

          Show
          arpitagarwal Arpit Agarwal added a comment - +1 pending jenkins for the v4 patch.
          Hide
          hualiu Hua Liu added a comment -

          Hi Arpit Agarwal

          I submitted the V4 patch a few hours ago but seems jenkins hasn't built it. I will re-submit tomorrow if jenkins still cannot kick in by tomorrow morning.

          Show
          hualiu Hua Liu added a comment - Hi Arpit Agarwal I submitted the V4 patch a few hours ago but seems jenkins hasn't built it. I will re-submit tomorrow if jenkins still cannot kick in by tomorrow morning.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 7m 28s trunk passed with JDK v1.8.0_74
          +1 compile 8m 1s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 10s trunk passed
          +1 mvnsite 2m 33s trunk passed
          +1 mvneclipse 0m 45s trunk passed
          +1 findbugs 5m 26s trunk passed
          +1 javadoc 3m 2s trunk passed with JDK v1.8.0_74
          +1 javadoc 4m 52s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 23s Maven dependency ordering for patch
          +1 mvninstall 2m 31s the patch passed
          +1 compile 9m 18s the patch passed with JDK v1.8.0_74
          +1 javac 9m 18s the patch passed
          +1 compile 7m 27s the patch passed with JDK v1.7.0_95
          +1 javac 7m 27s the patch passed
          -1 checkstyle 1m 5s root: patch generated 1 new + 83 unchanged - 0 fixed = 84 total (was 83)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 4m 1s the patch passed
          +1 javadoc 2m 15s the patch passed with JDK v1.8.0_74
          +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95
          -1 unit 8m 40s hadoop-common in the patch failed with JDK v1.8.0_74.
          -1 unit 69m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
          +1 unit 8m 37s hadoop-common in the patch passed with JDK v1.7.0_95.
          -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          220m 47s



          Reason Tests
          JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.hdfs.TestHFlush
            hadoop.hdfs.TestFileAppend
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
          JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791879/0004-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch
          JIRA Issue HDFS-9882
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 47c26453cb1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 391da36
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14737/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14737/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 7m 28s trunk passed with JDK v1.8.0_74 +1 compile 8m 1s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 10s trunk passed +1 mvnsite 2m 33s trunk passed +1 mvneclipse 0m 45s trunk passed +1 findbugs 5m 26s trunk passed +1 javadoc 3m 2s trunk passed with JDK v1.8.0_74 +1 javadoc 4m 52s trunk passed with JDK v1.7.0_95 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 2m 31s the patch passed +1 compile 9m 18s the patch passed with JDK v1.8.0_74 +1 javac 9m 18s the patch passed +1 compile 7m 27s the patch passed with JDK v1.7.0_95 +1 javac 7m 27s the patch passed -1 checkstyle 1m 5s root: patch generated 1 new + 83 unchanged - 0 fixed = 84 total (was 83) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 1s the patch passed +1 javadoc 2m 15s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 59s the patch passed with JDK v1.7.0_95 -1 unit 8m 40s hadoop-common in the patch failed with JDK v1.8.0_74. -1 unit 69m 40s hadoop-hdfs in the patch failed with JDK v1.8.0_74. +1 unit 8m 37s hadoop-common in the patch passed with JDK v1.7.0_95. -1 unit 58m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 220m 47s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.hdfs.TestHFlush   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791879/0004-HDFS-9882.Add-heartbeatsTotal-in-Datanode-metrics.patch JIRA Issue HDFS-9882 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 47c26453cb1c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 391da36 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14737/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14737/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14737/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed for 2.8.0. checkstyle issue can be ignored here given we follow the convention used by the class. Don't need a new test case.

          Thanks for the contribution Hua Liu.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed for 2.8.0. checkstyle issue can be ignored here given we follow the convention used by the class. Don't need a new test case. Thanks for the contribution Hua Liu .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9441 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9441/)
          HDFS-9882. Add heartbeatsTotal in Datanode metrics. (Contributed by Hua (arp: rev c2140d05efaf18b41caae8c61d9f6d668ab0e874)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9441 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9441/ ) HDFS-9882 . Add heartbeatsTotal in Datanode metrics. (Contributed by Hua (arp: rev c2140d05efaf18b41caae8c61d9f6d668ab0e874) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java

            People

            • Assignee:
              hualiu Hua Liu
              Reporter:
              hualiu Hua Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development