Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12409

Add metrics of execution time of different stages in EC recovery task

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha3
    • Fix Version/s: 3.0.0-beta1
    • Component/s: erasure-coding
    • Labels:
      None

      Description

      Admin can use more metrics to monitor EC recovery tasks, to get insights to tune recovery performance.

      1. HDFS-12409.01.patch
        5 kB
        Lei (Eddy) Xu
      2. HDFS-12409.00.patch
        5 kB
        Lei (Eddy) Xu

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        I've cherry-picked this to branch-3.0 for beta1.

        Show
        andrew.wang Andrew Wang added a comment - I've cherry-picked this to branch-3.0 for beta1.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12867/)
        HDFS-12409. Add metrics of execution time of different stages in EC (lei: rev 73aed34dffa5e79f6f819137b69054c1dee2d4dd)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12867/ ) HDFS-12409 . Add metrics of execution time of different stages in EC (lei: rev 73aed34dffa5e79f6f819137b69054c1dee2d4dd) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks Andrew Wang for the review.

        Committed to trunk.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks Andrew Wang for the review. Committed to trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 16s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 findbugs 1m 53s trunk passed
        +1 javadoc 0m 47s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 59s the patch passed
        +1 compile 0m 55s the patch passed
        +1 javac 0m 55s the patch passed
        +1 checkstyle 0m 38s the patch passed
        +1 mvnsite 1m 0s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 7s the patch passed
        +1 javadoc 0m 47s the patch passed
              Other Tests
        -1 unit 93m 11s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        123m 16s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestLeaseRecoveryStriped
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1d24ae41575f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f1d751b
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21100/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21100/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21100/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 16s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 57s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 47s trunk passed       Patch Compile Tests +1 mvninstall 0m 59s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 0s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 47s the patch passed       Other Tests -1 unit 93m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 123m 16s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestLeaseRecoveryStriped Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d24ae41575f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f1d751b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21100/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21100/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21100/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I retriggered this since I just committed HDFS-12417 and the host had bind exceptions.

        Show
        andrew.wang Andrew Wang added a comment - I retriggered this since I just committed HDFS-12417 and the host had bind exceptions.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 14m 36s trunk passed
        +1 compile 1m 1s trunk passed
        +1 checkstyle 0m 43s trunk passed
        +1 mvnsite 1m 9s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 0m 47s trunk passed
              Patch Compile Tests
        +1 mvninstall 1m 5s the patch passed
        +1 compile 0m 53s the patch passed
        +1 javac 0m 53s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 53s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 110m 42s hadoop-hdfs in the patch failed.
        -1 asflicense 0m 22s The patch generated 1 ASF License warnings.
        138m 54s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.TestFileAppendRestart
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestGetBlocks
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestModTime
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.server.namenode.TestReencryption
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 05a75bdea82b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 66be6c8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21082/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21082/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 36s trunk passed +1 compile 1m 1s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 9s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 47s trunk passed       Patch Compile Tests +1 mvninstall 1m 5s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 110m 42s hadoop-hdfs in the patch failed. -1 asflicense 0m 22s The patch generated 1 ASF License warnings. 138m 54s Reason Tests Failed junit tests hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestModTime   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.TestReencryption   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 05a75bdea82b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66be6c8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21082/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21082/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 51s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 1m 6s trunk passed
        +1 findbugs 1m 50s trunk passed
        +1 javadoc 0m 45s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 52s the patch passed
        +1 javac 0m 52s the patch passed
        -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68)
        +1 mvnsite 1m 6s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 1s the patch passed
        +1 javadoc 0m 41s the patch passed
              Other Tests
        -1 unit 108m 7s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        137m 38s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.server.namenode.TestReencryptionWithKMS
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestEncryptionZonesWithHA
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.server.namenode.TestReencryption
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.server.namenode.snapshot.TestSnapshot
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b3ccbb29aaf4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 738c2a9
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21079/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21079/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 51s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 6s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 45s trunk passed       Patch Compile Tests +1 mvninstall 0m 57s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68) +1 mvnsite 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 108m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 137m 38s Reason Tests Failed junit tests hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.namenode.TestReencryption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3ccbb29aaf4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 738c2a9 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21079/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21079/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Fix checkstyle warnings. The findbug warnings are not relevant and will fix in another JIRA.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Fix checkstyle warnings. The findbug warnings are not relevant and will fix in another JIRA.
        Hide
        andrew.wang Andrew Wang added a comment -

        We should also fix the checkstyles and check the failed tests are not related, I doubt they are.

        Show
        andrew.wang Andrew Wang added a comment - We should also fix the checkstyles and check the failed tests are not related, I doubt they are.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM thanks Eddy!

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM thanks Eddy!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 1m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 15m 23s trunk passed
        +1 compile 0m 56s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 1m 1s trunk passed
        -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings.
        +1 javadoc 0m 43s trunk passed
              Patch Compile Tests
        +1 mvninstall 1m 3s the patch passed
        +1 compile 0m 59s the patch passed
        +1 javac 0m 59s the patch passed
        -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68)
        +1 mvnsite 1m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 14s the patch passed
        +1 javadoc 0m 44s the patch passed
              Other Tests
        -1 unit 206m 9s hadoop-hdfs in the patch failed.
        +1 asflicense 1m 20s The patch does not generate ASF License warnings.
        237m 57s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.tools.TestDFSZKFailoverController
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestMissingBlocksAlert
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          hadoop.hdfs.TestFileAppendRestart
          hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.TestReplaceDatanodeOnFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.TestFsck
          hadoop.hdfs.server.datanode.TestDataNodeUUID
          hadoop.hdfs.TestDFSShell
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
          hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
          hadoop.hdfs.TestEncryptedTransfer
        Timed out junit tests org.apache.hadoop.hdfs.TestFileChecksum
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData
          org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2ba67db10c94 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8edc605
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21059/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21059/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 23s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 1s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 43s trunk passed       Patch Compile Tests +1 mvninstall 1m 3s the patch passed +1 compile 0m 59s the patch passed +1 javac 0m 59s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68) +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 14s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests -1 unit 206m 9s hadoop-hdfs in the patch failed. +1 asflicense 1m 20s The patch does not generate ASF License warnings. 237m 57s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestFileAppendRestart   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestEncryptedTransfer Timed out junit tests org.apache.hadoop.hdfs.TestFileChecksum   org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2ba67db10c94 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8edc605 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21059/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21059/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Add 3 metrics to measure the time spent on reading from sources, decoding, and writing to the targets.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Add 3 metrics to measure the time spent on reading from sources, decoding, and writing to the targets.

          People

          • Assignee:
            eddyxu Lei (Eddy) Xu
            Reporter:
            eddyxu Lei (Eddy) Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development