Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10917

Collect peer performance statistics on DataNode.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DataNodes already detect if replication pipeline operations are slow and log warnings. For the purpose of analysis, performance metrics are desirable. This proposes adding them on DataNodes.

      1. HDFS-10917.000.patch
        17 kB
        Xiaobing Zhou
      2. HDFS-10917.001.patch
        18 kB
        Xiaobing Zhou
      3. HDFS-10917.002.patch
        34 kB
        Xiaobing Zhou
      4. HDFS-10917.003.patch
        37 kB
        Xiaobing Zhou
      5. HDFS-10917.004.patch
        41 kB
        Xiaobing Zhou
      6. HDFS-10917.005.patch
        42 kB
        Xiaobing Zhou
      7. HDFS-10917-branch-2.005.patch
        42 kB
        Xiaobing Zhou
      8. HDFS-10917-branch-2.006.patch
        42 kB
        Xiaobing Zhou

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 17s HDFS-10917 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841383/HDFS-10917.000.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17736/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 17s HDFS-10917 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841383/HDFS-10917.000.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17736/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted v000 patch.

        • added these peer metrics:
          • sendPacketDownstreamTimeNanos
          • sendAckToUpstreamTimeNanos
          • writeDataToDiskTimeNanos
          • syncWriterOsCacheTimeNanos
          • flushOrSyncBlockTimeNanos
          • writeChecksumToDiskTimeNanos

        Will add tests for syncWriterOsCacheTimeNanos and writeDataToDiskTimeNanos in upcoming patches.

        Show
        xiaobingo Xiaobing Zhou added a comment - Posted v000 patch. added these peer metrics: sendPacketDownstreamTimeNanos sendAckToUpstreamTimeNanos writeDataToDiskTimeNanos syncWriterOsCacheTimeNanos flushOrSyncBlockTimeNanos writeChecksumToDiskTimeNanos Will add tests for syncWriterOsCacheTimeNanos and writeDataToDiskTimeNanos in upcoming patches.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        v001 added two more tests.

        • testSyncWriterOsCacheTimeNanosMetrics
        • testWriteDataToDiskTimeNanosMetrics
        Show
        xiaobingo Xiaobing Zhou added a comment - v001 added two more tests. testSyncWriterOsCacheTimeNanosMetrics testWriteDataToDiskTimeNanosMetrics
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 31s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 134 unchanged - 1 fixed = 150 total (was 135)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 48s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 68m 54s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        89m 40s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841580/HDFS-10917.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0aa7fb924323 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 51211a7
        Default Java 1.8.0_111
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17747/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17747/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 31s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 18s trunk passed -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 134 unchanged - 1 fixed = 150 total (was 135) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 68m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 89m 40s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841580/HDFS-10917.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0aa7fb924323 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 51211a7 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17747/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17747/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17747/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted patch v2 which scopes collecting stats of sendPacketDownstream by means of rolling average.

        Show
        xiaobingo Xiaobing Zhou added a comment - Posted patch v2 which scopes collecting stats of sendPacketDownstream by means of rolling average.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 14m 17s trunk passed
        +1 compile 9m 37s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 1m 56s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 9s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 22s the patch passed
        +1 compile 9m 14s the patch passed
        +1 javac 9m 14s the patch passed
        -0 checkstyle 1m 41s root: The patch generated 36 new + 624 unchanged - 2 fixed = 660 total (was 626)
        +1 mvnsite 1m 52s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 30s the patch passed
        +1 javadoc 1m 32s the patch passed
        -1 unit 7m 58s hadoop-common in the patch failed.
        +1 unit 62m 19s hadoop-hdfs in the patch passed.
        -1 asflicense 0m 33s The patch generated 1 ASF License warnings.
        125m 18s



        Reason Tests
        Failed junit tests hadoop.net.TestDNS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843992/HDFS-10917.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 1b9f13711332 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3583413
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17909/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17909/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 14m 17s trunk passed +1 compile 9m 37s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 9m 14s the patch passed +1 javac 9m 14s the patch passed -0 checkstyle 1m 41s root: The patch generated 36 new + 624 unchanged - 2 fixed = 660 total (was 626) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 30s the patch passed +1 javadoc 1m 32s the patch passed -1 unit 7m 58s hadoop-common in the patch failed. +1 unit 62m 19s hadoop-hdfs in the patch passed. -1 asflicense 0m 33s The patch generated 1 ASF License warnings. 125m 18s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843992/HDFS-10917.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 1b9f13711332 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3583413 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17909/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17909/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17909/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted v003 patch.

        1. fixed all check style issues.
        2. added some unit tests for peer metrics.
        3. did a bunch of refactoring.
        Show
        xiaobingo Xiaobing Zhou added a comment - Posted v003 patch. fixed all check style issues. added some unit tests for peer metrics. did a bunch of refactoring.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 2m 9s Maven dependency ordering for branch
        +1 mvninstall 12m 42s trunk passed
        +1 compile 9m 48s trunk passed
        +1 checkstyle 1m 39s trunk passed
        +1 mvnsite 1m 58s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 findbugs 3m 13s trunk passed
        +1 javadoc 1m 30s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 24s the patch passed
        +1 compile 9m 13s the patch passed
        +1 javac 9m 13s the patch passed
        -0 checkstyle 1m 38s root: The patch generated 3 new + 619 unchanged - 10 fixed = 622 total (was 629)
        +1 mvnsite 1m 55s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 31s the patch passed
        +1 javadoc 1m 32s the patch passed
        +1 unit 8m 29s hadoop-common in the patch passed.
        -1 unit 68m 48s hadoop-hdfs in the patch failed.
        -1 asflicense 0m 33s The patch generated 2 ASF License warnings.
        133m 2s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844348/HDFS-10917.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 226fbf27d0b0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 736f54b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17928/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17928/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 9s Maven dependency ordering for branch +1 mvninstall 12m 42s trunk passed +1 compile 9m 48s trunk passed +1 checkstyle 1m 39s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 3m 13s trunk passed +1 javadoc 1m 30s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 9m 13s the patch passed +1 javac 9m 13s the patch passed -0 checkstyle 1m 38s root: The patch generated 3 new + 619 unchanged - 10 fixed = 622 total (was 629) +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 32s the patch passed +1 unit 8m 29s hadoop-common in the patch passed. -1 unit 68m 48s hadoop-hdfs in the patch failed. -1 asflicense 0m 33s The patch generated 2 ASF License warnings. 133m 2s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844348/HDFS-10917.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 226fbf27d0b0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 736f54b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17928/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17928/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17928/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 2m 3s Maven dependency ordering for branch
        +1 mvninstall 13m 24s trunk passed
        +1 compile 10m 48s trunk passed
        +1 checkstyle 1m 42s trunk passed
        +1 mvnsite 1m 59s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 findbugs 3m 14s trunk passed
        +1 javadoc 1m 40s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 38s the patch passed
        +1 compile 10m 2s the patch passed
        +1 javac 10m 2s the patch passed
        -0 checkstyle 1m 39s root: The patch generated 3 new + 619 unchanged - 10 fixed = 622 total (was 629)
        +1 mvnsite 1m 53s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 38s the patch passed
        +1 javadoc 1m 37s the patch passed
        +1 unit 8m 18s hadoop-common in the patch passed.
        +1 unit 69m 34s hadoop-hdfs in the patch passed.
        -1 asflicense 0m 42s The patch generated 2 ASF License warnings.
        136m 48s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844349/HDFS-10917.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 01d0a2cfcfca 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 736f54b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17929/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17929/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17929/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17929/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 2m 3s Maven dependency ordering for branch +1 mvninstall 13m 24s trunk passed +1 compile 10m 48s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 1m 59s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 10m 2s the patch passed +1 javac 10m 2s the patch passed -0 checkstyle 1m 39s root: The patch generated 3 new + 619 unchanged - 10 fixed = 622 total (was 629) +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 38s the patch passed +1 javadoc 1m 37s the patch passed +1 unit 8m 18s hadoop-common in the patch passed. +1 unit 69m 34s hadoop-hdfs in the patch passed. -1 asflicense 0m 42s The patch generated 2 ASF License warnings. 136m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844349/HDFS-10917.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 01d0a2cfcfca 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 736f54b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17929/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17929/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/17929/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17929/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment - - edited

        Posted v4 patch.

        1. added license to two files.
        2. fix check style issues.
        3. added collecting thread local states.
        Show
        xiaobingo Xiaobing Zhou added a comment - - edited Posted v4 patch. added license to two files. fix check style issues. added collecting thread local states.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 12m 31s trunk passed
        +1 compile 9m 39s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 1m 58s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 16s trunk passed
        +1 javadoc 1m 33s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 24s the patch passed
        +1 compile 9m 13s the patch passed
        +1 javac 9m 13s the patch passed
        -0 checkstyle 1m 39s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687)
        +1 mvnsite 1m 54s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 31s the patch passed
        +1 javadoc 1m 34s the patch passed
        +1 unit 8m 34s hadoop-common in the patch passed.
        +1 unit 62m 48s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        124m 53s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844367/HDFS-10917.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 710448cef6db 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 56a13a6
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17931/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17931/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17931/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 31s trunk passed +1 compile 9m 39s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 16s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 9m 13s the patch passed +1 javac 9m 13s the patch passed -0 checkstyle 1m 39s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687) +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 34s the patch passed +1 unit 8m 34s hadoop-common in the patch passed. +1 unit 62m 48s hadoop-hdfs in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 124m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844367/HDFS-10917.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 710448cef6db 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 56a13a6 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17931/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17931/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17931/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Xiaobing Zhou for working on this. Patch v04 looks good to me. I just have a few questions:

        BlockReceiver.java
        1. Line 828: can you clarify the design behind tracking penultimate node only in a pipeline line [dn0, dn1, dn2]?
        It works well for example like: 5, 100, 105. What if we have latency of pipeline like: 5, 115, 15? Do we miss detecting non-penultimate slow datanodes?

        DataNodeMXBean.java
        2. Line 130: we are getting the average info of SendPacketDownstream when acting as penultimate (2nd to last) node.

        RollingAverages.java
        3. NIT: Line 63: should be .setNameFormat("RollingAverages-%d")

        Show
        xyao Xiaoyu Yao added a comment - Thanks Xiaobing Zhou for working on this. Patch v04 looks good to me. I just have a few questions: BlockReceiver.java 1. Line 828: can you clarify the design behind tracking penultimate node only in a pipeline line [dn0, dn1, dn2] ? It works well for example like: 5, 100, 105. What if we have latency of pipeline like: 5, 115, 15? Do we miss detecting non-penultimate slow datanodes? DataNodeMXBean.java 2. Line 130: we are getting the average info of SendPacketDownstream when acting as penultimate (2nd to last) node. RollingAverages.java 3. NIT: Line 63: should be .setNameFormat("RollingAverages-%d")
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Thanks Xiaoyu Yao. I posted v5 patch to address your comments #1, 2 and 3.

        Only tracking the latency of sending packet to the last node in pipeline is a conscious design choice.

        In the case of pipeline [dn0, dn1, dn2], 5ms latency from dn0 to dn1, 100ms from dn1 to dn2, NameNode claims dn2 is slow since it sees 100ms latency to dn2. Note that NameNode is not ware of pipeline structure in this context and only sees latency between two DataNodes.

        In another case of the same pipeline, 100ms latency from dn0 to dn1, 5ms from dn1 to dn2, NameNode will miss detecting dn1 being slow since it's not the last node. However the assumption is that in a busy enough cluster there are many other pipelines where dn1 is the last node, e.g. [dn3, dn4, dn1]. Also our tracking interval is relatively long enough (at least an hour) to improve the chances of the bad DataNodes being the last nodes in multiple pipelines.

        Show
        xiaobingo Xiaobing Zhou added a comment - Thanks Xiaoyu Yao . I posted v5 patch to address your comments #1, 2 and 3. Only tracking the latency of sending packet to the last node in pipeline is a conscious design choice. In the case of pipeline [dn0, dn1, dn2] , 5ms latency from dn0 to dn1, 100ms from dn1 to dn2, NameNode claims dn2 is slow since it sees 100ms latency to dn2. Note that NameNode is not ware of pipeline structure in this context and only sees latency between two DataNodes. In another case of the same pipeline, 100ms latency from dn0 to dn1, 5ms from dn1 to dn2, NameNode will miss detecting dn1 being slow since it's not the last node. However the assumption is that in a busy enough cluster there are many other pipelines where dn1 is the last node, e.g. [dn3, dn4, dn1] . Also our tracking interval is relatively long enough (at least an hour) to improve the chances of the bad DataNodes being the last nodes in multiple pipelines.
        Hide
        xyao Xiaoyu Yao added a comment -

        Xiaobing Zhou, thanks for updating the patch along with the rationale behind the sampling design of tracking send packet latency. I'm OK with the current penultimate node sampling design and think it might be easier to evaluate the effectiveness if we could make the sampling strategy configurable. We can do flexible sampling with a followup JIRA.

        +1 for v5 patch, pending Jenkins.

        Show
        xyao Xiaoyu Yao added a comment - Xiaobing Zhou , thanks for updating the patch along with the rationale behind the sampling design of tracking send packet latency. I'm OK with the current penultimate node sampling design and think it might be easier to evaluate the effectiveness if we could make the sampling strategy configurable. We can do flexible sampling with a followup JIRA. +1 for v5 patch, pending Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 12m 32s trunk passed
        +1 compile 9m 34s trunk passed
        +1 checkstyle 1m 40s trunk passed
        +1 mvnsite 1m 56s trunk passed
        +1 mvneclipse 0m 37s trunk passed
        +1 findbugs 3m 11s trunk passed
        +1 javadoc 1m 30s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 22s the patch passed
        +1 compile 9m 13s the patch passed
        +1 javac 9m 13s the patch passed
        -0 checkstyle 1m 40s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687)
        +1 mvnsite 1m 55s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 28s the patch passed
        +1 javadoc 1m 31s the patch passed
        +1 unit 8m 25s hadoop-common in the patch passed.
        +1 unit 66m 0s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        127m 38s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844477/HDFS-10917.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux dcbe9ea4e37d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22befbd
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17942/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17942/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17942/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 12m 32s trunk passed +1 compile 9m 34s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 30s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 9m 13s the patch passed +1 javac 9m 13s the patch passed -0 checkstyle 1m 40s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687) +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 31s the patch passed +1 unit 8m 25s hadoop-common in the patch passed. +1 unit 66m 0s hadoop-hdfs in the patch passed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 127m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844477/HDFS-10917.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux dcbe9ea4e37d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22befbd Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17942/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17942/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17942/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Xiaobing Zhou for the contribution. I've commit the latest patch to trunk.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Xiaobing Zhou for the contribution. I've commit the latest patch to trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 1m 52s Maven dependency ordering for branch
        +1 mvninstall 12m 43s trunk passed
        +1 compile 9m 41s trunk passed
        +1 checkstyle 1m 38s trunk passed
        +1 mvnsite 2m 1s trunk passed
        +1 mvneclipse 0m 35s trunk passed
        +1 findbugs 3m 14s trunk passed
        +1 javadoc 1m 31s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 25s the patch passed
        +1 compile 9m 15s the patch passed
        +1 javac 9m 15s the patch passed
        -0 checkstyle 1m 39s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687)
        +1 mvnsite 1m 55s the patch passed
        +1 mvneclipse 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 31s the patch passed
        +1 javadoc 1m 33s the patch passed
        -1 unit 7m 32s hadoop-common in the patch failed.
        +1 unit 65m 42s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 41s The patch does not generate ASF License warnings.
        128m 51s



        Reason Tests
        Failed junit tests hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10917
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844523/HDFS-10917.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux c9e5bd162ec6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e92a770
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17946/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17946/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17946/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17946/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 52s Maven dependency ordering for branch +1 mvninstall 12m 43s trunk passed +1 compile 9m 41s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 2m 1s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 3m 14s trunk passed +1 javadoc 1m 31s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 9m 15s the patch passed +1 javac 9m 15s the patch passed -0 checkstyle 1m 39s root: The patch generated 1 new + 678 unchanged - 9 fixed = 679 total (was 687) +1 mvnsite 1m 55s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 31s the patch passed +1 javadoc 1m 33s the patch passed -1 unit 7m 32s hadoop-common in the patch failed. +1 unit 65m 42s hadoop-hdfs in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 128m 51s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10917 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844523/HDFS-10917.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c9e5bd162ec6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e92a770 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17946/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17946/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17946/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17946/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11033 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11033/)
        HDFS-10917. Collect peer performance statistics on DataNode. Contributed (xyao: rev 4e9029653dfa7a803d73c173cb7044f7e0dc1eb1)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsJsonBuilder.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodePeerMetrics.java
        • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/RollingAverages.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
        • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestRollingAverages.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11033 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11033/ ) HDFS-10917 . Collect peer performance statistics on DataNode. Contributed (xyao: rev 4e9029653dfa7a803d73c173cb7044f7e0dc1eb1) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeMXBean.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsJsonBuilder.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodePeerMetrics.java (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/RollingAverages.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodePeerMetrics.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableRatesWithAggregation.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestRollingAverages.java
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted branch-2 patch.

        Show
        xiaobingo Xiaobing Zhou added a comment - Posted branch-2 patch.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        v006 made some changes by removing unnecessary queue allocations.

        224	      LinkedBlockingDeque<SumAndCount> deque = new LinkedBlockingDeque<SumAndCount>(
        225	          numWindows);
        
        Show
        xiaobingo Xiaobing Zhou added a comment - v006 made some changes by removing unnecessary queue allocations. 224 LinkedBlockingDeque<SumAndCount> deque = new LinkedBlockingDeque<SumAndCount>( 225 numWindows);
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks Xiaobing. I've committed this to branch-2.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks Xiaobing. I've committed this to branch-2.

          People

          • Assignee:
            xiaobingo Xiaobing Zhou
            Reporter:
            xiaobingo Xiaobing Zhou
          • Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development