Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10645

Make block report size as a metric and add this metric to datanode web ui

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: datanode, ui
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Record block report size as a metric and show it on datanode UI. It's important for administrators to know the bottleneck of block report, and the metric is also a good tuning metric.

      1. HDFS-10645.001.patch
        8 kB
        Yuanbo Liu
      2. HDFS-10645.002.patch
        7 kB
        Yuanbo Liu
      3. HDFS-10645.003.patch
        7 kB
        Yuanbo Liu
      4. HDFS-10645.004.patch
        10 kB
        Yuanbo Liu
      5. HDFS-10645.005.patch
        12 kB
        Yuanbo Liu
      6. HDFS-10645.006.patch
        12 kB
        Yuanbo Liu
      7. HDFS-10645.007.patch
        11 kB
        Yuanbo Liu
      8. HDFS-10645.008.patch
        11 kB
        Yuanbo Liu
      9. HDFS-10645.009.patch
        11 kB
        Yuanbo Liu
      10. HDFS-10645-branch-2.001.patch
        11 kB
        Akira Ajisaka
      11. Selection_047.png
        14 kB
        Yuanbo Liu
      12. Selection_048.png
        15 kB
        Yuanbo Liu

        Issue Links

          Activity

          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your committing!

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your committing!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to branch-2. Thanks Yuanbo Liu for the continuous work!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to branch-2. Thanks Yuanbo Liu for the continuous work!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Whitespace issue and test failures are not related to the patch. Committing this to branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Whitespace issue and test failures are not related to the patch. Committing this to branch-2.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 46s branch-2 passed
          +1 compile 5m 34s branch-2 passed with JDK v1.8.0_101
          +1 compile 6m 27s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 1m 23s branch-2 passed
          +1 mvnsite 1m 52s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 3m 40s branch-2 passed
          +1 javadoc 1m 41s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 2m 43s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 39s the patch passed
          +1 compile 6m 2s the patch passed with JDK v1.8.0_101
          +1 javac 6m 2s the patch passed
          +1 compile 6m 42s the patch passed with JDK v1.7.0_111
          +1 javac 6m 42s the patch passed
          +1 checkstyle 1m 23s the patch passed
          +1 mvnsite 1m 53s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 4m 12s the patch passed
          +1 javadoc 2m 3s the patch passed with JDK v1.8.0_101
          +1 javadoc 2m 39s the patch passed with JDK v1.7.0_111
          -1 unit 8m 48s hadoop-common in the patch failed with JDK v1.7.0_111.
          -1 unit 65m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          221m 7s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
          JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_111 Failed junit tests hadoop.security.TestGroupsCaching
            hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824508/HDFS-10645-branch-2.001.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 7032e19e442d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 50a8b0e
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16491/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16491/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 12s Maven dependency ordering for branch +1 mvninstall 6m 46s branch-2 passed +1 compile 5m 34s branch-2 passed with JDK v1.8.0_101 +1 compile 6m 27s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 1m 23s branch-2 passed +1 mvnsite 1m 52s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 3m 40s branch-2 passed +1 javadoc 1m 41s branch-2 passed with JDK v1.8.0_101 +1 javadoc 2m 43s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 6m 2s the patch passed with JDK v1.8.0_101 +1 javac 6m 2s the patch passed +1 compile 6m 42s the patch passed with JDK v1.7.0_111 +1 javac 6m 42s the patch passed +1 checkstyle 1m 23s the patch passed +1 mvnsite 1m 53s the patch passed +1 mvneclipse 0m 30s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 4m 12s the patch passed +1 javadoc 2m 3s the patch passed with JDK v1.8.0_101 +1 javadoc 2m 39s the patch passed with JDK v1.7.0_111 -1 unit 8m 48s hadoop-common in the patch failed with JDK v1.7.0_111. -1 unit 65m 24s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 221m 7s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider JDK v1.8.0_101 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_111 Failed junit tests hadoop.security.TestGroupsCaching   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824508/HDFS-10645-branch-2.001.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 7032e19e442d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 50a8b0e Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_111.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16491/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16491/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka It's very kind of you to rebase the patch for branch-2. Thanks very much !

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka It's very kind of you to rebase the patch for branch-2. Thanks very much !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10305 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10305/)
          HDFS-10645. Make block report size as a metric and add this metric to (aajisaka: rev 8179f9a493c1b26deb6b1bffacd6a829586b7f98)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10305 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10305/ ) HDFS-10645 . Make block report size as a metric and add this metric to (aajisaka: rev 8179f9a493c1b26deb6b1bffacd6a829586b7f98) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk.
          I had to rebase the patch for branch-2, so attaching a patch for branch-2. I'll commit this to branch-2 if Jenkins says okay.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk. I had to rebase the patch for branch-2, so attaching a patch for branch-2. I'll commit this to branch-2 if Jenkins says okay.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your review.
          I've tested the failures TestBootstrapStandby and TestCrcCorruption in my local environment and they all pass.
          So I think those test failures are not related to my change.
          Thanks again for your time.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your review. I've tested the failures TestBootstrapStandby and TestCrcCorruption in my local environment and they all pass. So I think those test failures are not related to my change. Thanks again for your time.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 6m 50s trunk passed
          +1 checkstyle 1m 23s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 6s trunk passed
          +1 javadoc 1m 52s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 32s the patch passed
          +1 compile 7m 35s the patch passed
          +1 javac 7m 35s the patch passed
          +1 checkstyle 1m 32s the patch passed
          +1 mvnsite 2m 15s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 49s the patch passed
          +1 javadoc 1m 49s the patch passed
          +1 unit 9m 38s hadoop-common in the patch passed.
          -1 unit 77m 18s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          130m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
            hadoop.hdfs.TestCrcCorruption



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822937/HDFS-10645.009.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux cb3de08e707f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ffe1fff
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16437/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16437/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 6m 50s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 52s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 7m 35s the patch passed +1 javac 7m 35s the patch passed +1 checkstyle 1m 32s the patch passed +1 mvnsite 2m 15s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 49s the patch passed +1 javadoc 1m 49s the patch passed +1 unit 9m 38s hadoop-common in the patch passed. -1 unit 77m 18s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 130m 27s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822937/HDFS-10645.009.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux cb3de08e707f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ffe1fff Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16437/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16437/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 pending Jenkins. Thanks Yuanbo Liu.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 pending Jenkins. Thanks Yuanbo Liu .
          Hide
          yuanbo Yuanbo Liu added a comment -

          Uploaded v9 patch to address Akira Ajisaka's comment.

          Show
          yuanbo Yuanbo Liu added a comment - Uploaded v9 patch to address Akira Ajisaka 's comment.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Yes I agree with you, since maxDataLength is part of BPServiceActorInfo, we should document high level metric.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Yes I agree with you, since maxDataLength is part of BPServiceActorInfo , we should document high level metric.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thank you for updating the patch, Yuanbo Liu.

          Metrics.md
          +| `maxBlockReportSize` | Max size of block report |
          

          Rethinking this, maxBlockReportSize and maxDataLength are in BPServiceActorInfo, so I'm thinking BPServiceActorInfo should be documented instead.

          Show
          ajisakaa Akira Ajisaka added a comment - Thank you for updating the patch, Yuanbo Liu . Metrics.md +| `maxBlockReportSize` | Max size of block report | Rethinking this, maxBlockReportSize and maxDataLength are in BPServiceActorInfo, so I'm thinking BPServiceActorInfo should be documented instead.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Could you please take a look at my patch? I believe that those failures are not related to my changes.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Could you please take a look at my patch? I believe that those failures are not related to my changes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 42s Maven dependency ordering for branch
          +1 mvninstall 9m 16s trunk passed
          +1 compile 7m 26s trunk passed
          +1 checkstyle 1m 30s trunk passed
          +1 mvnsite 1m 58s trunk passed
          +1 mvneclipse 0m 29s trunk passed
          +1 findbugs 3m 8s trunk passed
          +1 javadoc 1m 52s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 34s the patch passed
          +1 compile 7m 2s the patch passed
          +1 javac 7m 2s the patch passed
          +1 checkstyle 1m 29s the patch passed
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 39s the patch passed
          +1 javadoc 1m 43s the patch passed
          -1 unit 20m 32s hadoop-common in the patch failed.
          -1 unit 78m 46s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          145m 31s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.security.ssl.TestReloadingX509TrustManager
            hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
            hadoop.tracing.TestTracing
            hadoop.security.TestRefreshUserMappings
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822225/HDFS-10645.008.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux de5af17410ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 438a9f0
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16325/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16325/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16325/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16325/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 9m 16s trunk passed +1 compile 7m 26s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 52s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 34s the patch passed +1 compile 7m 2s the patch passed +1 javac 7m 2s the patch passed +1 checkstyle 1m 29s the patch passed +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 39s the patch passed +1 javadoc 1m 43s the patch passed -1 unit 20m 32s hadoop-common in the patch failed. -1 unit 78m 46s hadoop-hdfs in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 145m 31s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.security.ssl.TestReloadingX509TrustManager   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.tracing.TestTracing   hadoop.security.TestRefreshUserMappings Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822225/HDFS-10645.008.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux de5af17410ec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 438a9f0 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16325/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16325/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16325/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16325/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Don't know why resume button doesn't work, upload v8 patch to trigger the jenkins.

          Show
          yuanbo Yuanbo Liu added a comment - Don't know why resume button doesn't work, upload v8 patch to trigger the jenkins.
          Hide
          yuanbo Yuanbo Liu added a comment -

          I don't think these failures are related to my change. Resume the progress to get new result.

          Show
          yuanbo Yuanbo Liu added a comment - I don't think these failures are related to my change. Resume the progress to get new result.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 8m 20s Maven dependency ordering for branch
          +1 mvninstall 8m 14s trunk passed
          +1 compile 7m 56s trunk passed
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 1m 56s trunk passed
          +1 mvneclipse 0m 30s trunk passed
          +1 findbugs 3m 59s trunk passed
          +1 javadoc 1m 54s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 50s the patch passed
          +1 compile 8m 14s the patch passed
          +1 javac 8m 14s the patch passed
          +1 checkstyle 1m 35s the patch passed
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 58s the patch passed
          +1 javadoc 1m 49s the patch passed
          -1 unit 7m 46s hadoop-common in the patch failed.
          -1 unit 69m 1s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          132m 47s



          Reason Tests
          Failed junit tests hadoop.log.TestLogLevel
            hadoop.hdfs.qjournal.TestNNWithQJM
            hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.TestDecommissionWithStriped
            hadoop.hdfs.TestRenameWhileOpen



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821769/HDFS-10645.007.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux f9c455d757e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d28c2d9
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16303/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16303/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16303/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16303/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 8m 20s Maven dependency ordering for branch +1 mvninstall 8m 14s trunk passed +1 compile 7m 56s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 56s trunk passed +1 mvneclipse 0m 30s trunk passed +1 findbugs 3m 59s trunk passed +1 javadoc 1m 54s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 50s the patch passed +1 compile 8m 14s the patch passed +1 javac 8m 14s the patch passed +1 checkstyle 1m 35s the patch passed +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 58s the patch passed +1 javadoc 1m 49s the patch passed -1 unit 7m 46s hadoop-common in the patch failed. -1 unit 69m 1s hadoop-hdfs in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 132m 47s Reason Tests Failed junit tests hadoop.log.TestLogLevel   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestRenameWhileOpen Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821769/HDFS-10645.007.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux f9c455d757e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d28c2d9 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16303/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16303/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16303/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16303/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Sure, uploaded v7 patch to address it.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Sure, uploaded v7 patch to address it.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Mostly looks good to me. Hi Yuanbo Liu, would you rebase the patch?

          Show
          ajisakaa Akira Ajisaka added a comment - Mostly looks good to me. Hi Yuanbo Liu , would you rebase the patch?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 7m 16s trunk passed
          +1 compile 7m 6s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 51s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 1m 43s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 6m 59s the patch passed
          +1 javac 6m 59s the patch passed
          +1 checkstyle 1m 25s the patch passed
          +1 mvnsite 1m 48s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 45s the patch passed
          +1 javadoc 1m 50s the patch passed
          +1 unit 8m 21s hadoop-common in the patch passed.
          +1 unit 72m 14s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          123m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820916/HDFS-10645.006.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux b13801e63d36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 204a205
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16248/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16248/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 7m 16s trunk passed +1 compile 7m 6s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 6m 59s the patch passed +1 javac 6m 59s the patch passed +1 checkstyle 1m 25s the patch passed +1 mvnsite 1m 48s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 45s the patch passed +1 javadoc 1m 50s the patch passed +1 unit 8m 21s hadoop-common in the patch passed. +1 unit 72m 14s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 123m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820916/HDFS-10645.006.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux b13801e63d36 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 204a205 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16248/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16248/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Thanks for your quick response!
          Yes you're right, the getter method can be marked as package private, I will upload v6 patch soon.

          Show
          yuanbo Yuanbo Liu added a comment - Thanks for your quick response! Yes you're right, the getter method can be marked as package private, I will upload v6 patch soon.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Can we change the getter method to package private instead of the local variable?

          Show
          ajisakaa Akira Ajisaka added a comment - Can we change the getter method to package private instead of the local variable?
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your comments.

          DNConf#maxDataLength() can be package private

          I tried it before, but checkstyle policy would alert warning, and voted -1 for this code. So I change it to private property.

          1. The line is inserted wrongly. Would you insert the line as follows?

          Sorry for this mistake, I will change it in V6 patch.
          Hope to get your thoughts, Thanks for your time.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your comments. DNConf#maxDataLength() can be package private I tried it before, but checkstyle policy would alert warning, and voted -1 for this code. So I change it to private property. 1. The line is inserted wrongly. Would you insert the line as follows? Sorry for this mistake, I will change it in V6 patch. Hope to get your thoughts, Thanks for your time.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thank you for updating the patch, Yuanbo Liu. Some comments.

          @@ -113,6 +114,7 @@
                 <td>{reservedSpace|fmt_bytes}</td>
                 <td>{reservedSpaceForReplicas|fmt_bytes}</td>
                 <td>{numBlocks}</td>
          +      <td>{maxBlockReportSize|fmt_bytes} ({maxDataLength|fmt_bytes})</td>
               </tr>
             {/dn.VolumeInfo}
           </script>
          

          1. The line is inserted wrongly. Would you insert the line as follows?

                <td>{#helper_relative_time value="{LastBlockReport}"/}</td>
          +     <td>{maxBlockReportSize|fmt_bytes} ({maxDataLength|fmt_bytes})</td>
              </tr>
          

          2. DNConf#maxDataLength() can be package private.
          3. I don't think the following comment is necessary. This is obvious for me.

              // get the value of maxDataLength.
              this.maxDataLength = dnConf.getMaxDataLength();
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thank you for updating the patch, Yuanbo Liu . Some comments. @@ -113,6 +114,7 @@ <td>{reservedSpace|fmt_bytes}</td> <td>{reservedSpaceForReplicas|fmt_bytes}</td> <td>{numBlocks}</td> + <td>{maxBlockReportSize|fmt_bytes} ({maxDataLength|fmt_bytes})</td> </tr> {/dn.VolumeInfo} </script> 1. The line is inserted wrongly. Would you insert the line as follows? <td>{#helper_relative_time value= "{LastBlockReport}" /}</td> + <td>{maxBlockReportSize|fmt_bytes} ({maxDataLength|fmt_bytes})</td> </tr> 2. DNConf#maxDataLength() can be package private. 3. I don't think the following comment is necessary. This is obvious for me. // get the value of maxDataLength. this .maxDataLength = dnConf.getMaxDataLength();
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 38s trunk passed
          +1 compile 6m 46s trunk passed
          +1 checkstyle 1m 25s trunk passed
          +1 mvnsite 1m 45s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 27s the patch passed
          +1 compile 6m 41s the patch passed
          +1 javac 6m 41s the patch passed
          +1 checkstyle 1m 24s the patch passed
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 20s the patch passed
          +1 javadoc 1m 40s the patch passed
          +1 unit 7m 10s hadoop-common in the patch passed.
          -1 unit 62m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          109m 35s



          Reason Tests
          Failed junit tests hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820664/HDFS-10645.005.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle
          uname Linux 950f1e8d98bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8d06bda
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16233/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16233/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16233/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 6m 46s trunk passed +1 checkstyle 1m 25s trunk passed +1 mvnsite 1m 45s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 6m 41s the patch passed +1 javac 6m 41s the patch passed +1 checkstyle 1m 24s the patch passed +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 20s the patch passed +1 javadoc 1m 40s the patch passed +1 unit 7m 10s hadoop-common in the patch passed. -1 unit 62m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 109m 35s Reason Tests Failed junit tests hadoop.cli.TestHDFSCLI Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12820664/HDFS-10645.005.patch JIRA Issue HDFS-10645 Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle uname Linux 950f1e8d98bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8d06bda Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16233/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16233/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16233/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your comments
          I changed a bit code because of HDFS-10301. But I don't think it effects the calculation of block size.
          Upload v5 for review. Thank you again for your suggestion, that's very helpful !

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your comments I changed a bit code because of HDFS-10301 . But I don't think it effects the calculation of block size. Upload v5 for review. Thank you again for your suggestion, that's very helpful !
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'm thinking we can add maxDataLength in DNConf as follows:

          DNConf.java (constructor)
              maxDataLength = conf.getInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
                  CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT);
          
          BPServiceActor.java (constructor)
              this.maxDataLength = dnConf.maxDataLength;
          

          That way we can avoid NPE in the unit test because the constructor of DNConf is not called in the test. Sorry for late response.

          Show
          ajisakaa Akira Ajisaka added a comment - I'm thinking we can add maxDataLength in DNConf as follows: DNConf.java (constructor) maxDataLength = conf.getInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH, CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT); BPServiceActor.java (constructor) this .maxDataLength = dnConf.maxDataLength; That way we can avoid NPE in the unit test because the constructor of DNConf is not called in the test. Sorry for late response.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your comments and your patience.

          When setting maxDataLength, would you use dnConf instead of dn.getConf()? Null check is not needed.

          Agree to use dnConf instead of dn.getConf(). But the null check can not be deleted. I've found some code from TestDatanodeRegister#setUpsetUp

              DataNode mockDN = mock(DataNode.class);
              doReturn(true).when(mockDN).shouldRun();
              doReturn(mockDnConf).when(mockDN).getDnConf();
          

          it uses mock to test some functions. I've no knowledge about mock in unit test, but it will cause nullpoint exception when I write code in BPServiceActor like this:

              this.maxDataLength =
                  dnConf.getConf().getInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
                  CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT);
          

          I hope to get your thoughts before I upload v5 patch.
          Thank you again for your time.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your comments and your patience. When setting maxDataLength, would you use dnConf instead of dn.getConf()? Null check is not needed. Agree to use dnConf instead of dn.getConf(). But the null check can not be deleted. I've found some code from TestDatanodeRegister#setUpsetUp DataNode mockDN = mock(DataNode.class); doReturn( true ).when(mockDN).shouldRun(); doReturn(mockDnConf).when(mockDN).getDnConf(); it uses mock to test some functions. I've no knowledge about mock in unit test, but it will cause nullpoint exception when I write code in BPServiceActor like this: this .maxDataLength = dnConf.getConf().getInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH, CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT); I hope to get your thoughts before I upload v5 patch. Thank you again for your time.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          In addition, would you update Metrics.md to document the new metrics?

          Show
          ajisakaa Akira Ajisaka added a comment - In addition, would you update Metrics.md to document the new metrics?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yuanbo Liu for the update! I'm thinking we're getting close. Some comments:

          1. When setting maxDataLength, would you use dnConf instead of dn.getConf()? Null check is not needed.
          2. maxDataLength should be final.
          3. After HDFS-10287, we can use try-with-resources statement for MiniDFSCluster. Would you use the statement instead of shutting down the cluster in finally clause?
          4. In Apache Hadoop, Jackson is commonly used for reading/writing json. Would you use Jackson instead of Gson in the test code?
          5. Would you reverse the order of the arguments in the following assertion?
                  assertEquals("maxDataLength should be exactly "
                      + "the same value of ipc.maximum.data.length",
                      maxDataLength,
                      confMaxDataLength);
            

            Actual value is after expected value in Assert.assertEquals(message, expected, actual).

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yuanbo Liu for the update! I'm thinking we're getting close. Some comments: When setting maxDataLength , would you use dnConf instead of dn.getConf() ? Null check is not needed. maxDataLength should be final. After HDFS-10287 , we can use try-with-resources statement for MiniDFSCluster. Would you use the statement instead of shutting down the cluster in finally clause? In Apache Hadoop, Jackson is commonly used for reading/writing json. Would you use Jackson instead of Gson in the test code? Would you reverse the order of the arguments in the following assertion? assertEquals( "maxDataLength should be exactly " + "the same value of ipc.maximum.data.length" , maxDataLength, confMaxDataLength); Actual value is after expected value in Assert.assertEquals(message, expected, actual).
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 0s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 57s trunk passed
          +1 javadoc 0m 59s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 50s the patch passed
          +1 javac 0m 50s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 5s the patch passed
          +1 javadoc 1m 0s the patch passed
          +1 unit 62m 34s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          84m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819267/HDFS-10645.004.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c7b9a1973b6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 521f343
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16132/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16132/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 0s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 57s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 5s the patch passed +1 javadoc 1m 0s the patch passed +1 unit 62m 34s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 84m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819267/HDFS-10645.004.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c7b9a1973b6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 521f343 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16132/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16132/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          uploaded v4 patch

          Show
          yuanbo Yuanbo Liu added a comment - uploaded v4 patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 42s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 55s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 48s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 52s the patch passed
          +1 javadoc 0m 53s the patch passed
          -1 unit 78m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          100m 12s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestEditLog



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 73812f787435 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e340064
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16129/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16129/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16129/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 42s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 52s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 78m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 100m 12s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestEditLog Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 73812f787435 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e340064 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16129/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16129/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16129/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 7s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 4s trunk passed
          +1 mvninstall 1m 1s the patch passed
          +1 compile 0m 55s the patch passed
          +1 javac 0m 55s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 61m 10s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          83m 40s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
            hadoop.hdfs.qjournal.client.TestQuorumJournalManager
            hadoop.hdfs.server.blockmanagement.TestBlockManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2d2f2f261fb5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e340064
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16123/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16123/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16123/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 7s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 4s trunk passed +1 mvninstall 1m 1s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 61m 10s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 83m 40s Reason Tests Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.qjournal.client.TestQuorumJournalManager   hadoop.hdfs.server.blockmanagement.TestBlockManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2d2f2f261fb5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e340064 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16123/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16123/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16123/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your good guidance. I will create a new patch later today.

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your good guidance. I will create a new patch later today.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 40s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 54s the patch passed
          -1 unit 58m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 2s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ef4933cd339e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e340064
          Default Java 1.8.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16120/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16120/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16120/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 40s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 54s the patch passed -1 unit 58m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 2s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819012/HDFS-10645.003.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ef4933cd339e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e340064 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/16120/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16120/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16120/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -
              // need to keep maxDataLength up-to-date, this is a configurable property.
              this.maxDataLength = dn.getConf().getInt(
                  CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
                  CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT);
          

          The parameter is configurable but not reconfigurable, so it's sufficient to set only once in the constructor.

          Show
          ajisakaa Akira Ajisaka added a comment - // need to keep maxDataLength up-to-date, this is a configurable property. this .maxDataLength = dn.getConf().getInt( CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH, CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT); The parameter is configurable but not reconfigurable, so it's sufficient to set only once in the constructor.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thank you for the update.

          1. blockReportSizes must be synchronized. I recommend to use Collections.synchronizedSortedSet(new TreeSet<>());
          2. Would you make blockReportSizes final? That way we can avoid null check in getMaxBlockReportSize.
          3. Would you use !blockReportSizes.isEmpty() instead of blockReportSizes.size() > 0 ?
          4. In the regression test, would you check the followings?
          • the max block report size is greater than zero
          • the max data length is equal to 64MB
          Show
          ajisakaa Akira Ajisaka added a comment - Thank you for the update. blockReportSizes must be synchronized. I recommend to use Collections.synchronizedSortedSet(new TreeSet<>()) ; Would you make blockReportSizes final? That way we can avoid null check in getMaxBlockReportSize. Would you use !blockReportSizes.isEmpty() instead of blockReportSizes.size() > 0 ? In the regression test, would you check the followings? the max block report size is greater than zero the max data length is equal to 64MB
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your comments, they are very helpful!
          Uploaded v3 patch

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your comments, they are very helpful! Uploaded v3 patch
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yuanbo Liu for updating the patch. Some comments:

          1. Would you remove glob pattern in imports?
          2. I'm thinking SortedSet.last() is better than Collections.max(HashSet) because we use only the maximum value. Sorry for the back and forth.
          3. Can we reuse the set to avoid creating instances for each block report?
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yuanbo Liu for updating the patch. Some comments: Would you remove glob pattern in imports? I'm thinking SortedSet.last() is better than Collections.max(HashSet) because we use only the maximum value. Sorry for the back and forth. Can we reuse the set to avoid creating instances for each block report?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 53s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 22s the patch passed
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 51s the patch passed
          +1 unit 58m 36s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          77m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818789/HDFS-10645.002.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 55715358c472 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fe20494
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16091/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16091/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 51s the patch passed +1 unit 58m 36s hadoop-hdfs in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 77m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818789/HDFS-10645.002.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 55715358c472 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fe20494 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16091/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16091/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment -

          Akira Ajisaka Thanks for your comments. I've uploaded v2 patch

          Show
          yuanbo Yuanbo Liu added a comment - Akira Ajisaka Thanks for your comments. I've uploaded v2 patch
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 for the idea. Some comments from me:

          1. Can we use Set<Integer> for blockReportSizes instead of int[]? That way we can get the max value by just calling Collections.max. (and need to synchronize the set)
          2. Would you remove unused imports from TestDataNodeMXBean?
          3. Would you remove the following unnecessarily comments from the test?

                LOG.info("yuanbo print here " + dn.getBPServiceActorInfo());
          

          Logging is good, but do not print your name in any source code.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 for the idea. Some comments from me: 1. Can we use Set<Integer> for blockReportSizes instead of int[]? That way we can get the max value by just calling Collections.max . (and need to synchronize the set) 2. Would you remove unused imports from TestDataNodeMXBean? 3. Would you remove the following unnecessarily comments from the test? LOG.info( "yuanbo print here " + dn.getBPServiceActorInfo()); Logging is good, but do not print your name in any source code.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 47s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 31 unchanged - 0 fixed = 38 total (was 31)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 57s the patch passed
          -1 unit 73m 27s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          92m 53s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
            hadoop.hdfs.server.datanode.TestDatanodeRegister
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
            hadoop.hdfs.TestSafeMode



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818521/HDFS-10645.001.patch
          JIRA Issue HDFS-10645
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 06503efec006 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5b4a708
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16079/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16079/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16079/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16079/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 31 unchanged - 0 fixed = 38 total (was 31) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 73m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 92m 53s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport   hadoop.hdfs.server.datanode.TestDatanodeRegister   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.TestSafeMode Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818521/HDFS-10645.001.patch JIRA Issue HDFS-10645 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 06503efec006 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5b4a708 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16079/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16079/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16079/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16079/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yuanbo Yuanbo Liu added a comment - - edited

          Weiwei Yang Good point. I submitted a new snapshot(Selection_048.png) and v1 patch.

          Show
          yuanbo Yuanbo Liu added a comment - - edited Weiwei Yang Good point. I submitted a new snapshot(Selection_048.png) and v1 patch.
          Hide
          cheersyang Weiwei Yang added a comment -

          It would be good to expose the BR size in datanode metrics, so it would be possible for users to monitor the size (vs protobuf limit), build some alerts when it exceed a threshold. It is really annoying when DN failed to send BR when it gets too big, and it's hard to figure out why. Add it on UI is also good, but need to think how to display it, e.g

          Last Block Report Size (Max Size)
          32mb (64mb)
          Show
          cheersyang Weiwei Yang added a comment - It would be good to expose the BR size in datanode metrics, so it would be possible for users to monitor the size (vs protobuf limit), build some alerts when it exceed a threshold. It is really annoying when DN failed to send BR when it gets too big, and it's hard to figure out why. Add it on UI is also good, but need to think how to display it, e.g Last Block Report Size (Max Size) 32mb (64mb)
          Hide
          yuanbo Yuanbo Liu added a comment - - edited

          If the cluster grows big enough, it will hit this error:

          org.apache.hadoop.ipc.RemoteException: java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
          

          Apparently the block report size exceed the limit of PB, and the blocks in the data directory will be marked as unavailable in namespace. This is a bad sign for the cluster despite of 3 replications. It's will be better if the administrators get the "block report size" in time. So I propose to add this metric to datanode web ui.

          Show
          yuanbo Yuanbo Liu added a comment - - edited If the cluster grows big enough, it will hit this error: org.apache.hadoop.ipc.RemoteException: java.lang.IllegalStateException: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit. Apparently the block report size exceed the limit of PB, and the blocks in the data directory will be marked as unavailable in namespace. This is a bad sign for the cluster despite of 3 replications. It's will be better if the administrators get the "block report size" in time. So I propose to add this metric to datanode web ui.

            People

            • Assignee:
              yuanbo Yuanbo Liu
              Reporter:
              yuanbo Yuanbo Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development