Lei (Eddy) Xu added a comment - 08/Sep/17 21:37 Add 3 metrics to measure the time spent on reading from sources, decoding, and writing to the targets.

Hadoop QA added a comment - 09/Sep/17 01:44 -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 23s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 1s trunk passed -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 43s trunk passed       Patch Compile Tests +1 mvninstall 1m 3s the patch passed +1 compile 0m 59s the patch passed +1 javac 0m 59s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68) +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 14s the patch passed +1 javadoc 0m 44s the patch passed       Other Tests -1 unit 206m 9s hadoop-hdfs in the patch failed. +1 asflicense 1m 20s The patch does not generate ASF License warnings. 237m 57s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestMissingBlocksAlert   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestFileAppendRestart   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestReplaceDatanodeOnFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFsck   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.hdfs.TestEncryptedTransfer Timed out junit tests org.apache.hadoop.hdfs.TestFileChecksum   org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingDeletedData Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2ba67db10c94 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8edc605 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21059/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21059/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21059/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Andrew Wang added a comment - 11/Sep/17 18:18 +1 LGTM thanks Eddy!

Andrew Wang added a comment - 11/Sep/17 18:18 We should also fix the checkstyles and check the failed tests are not related, I doubt they are.

Lei (Eddy) Xu added a comment - 11/Sep/17 23:09 Fix checkstyle warnings. The findbug warnings are not relevant and will fix in another JIRA.

Hadoop QA added a comment - 12/Sep/17 00:40 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 51s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 6s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 45s trunk passed       Patch Compile Tests +1 mvninstall 0m 57s the patch passed +1 compile 0m 52s the patch passed +1 javac 0m 52s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 68 unchanged - 0 fixed = 71 total (was 68) +1 mvnsite 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 1s the patch passed +1 javadoc 0m 41s the patch passed       Other Tests -1 unit 108m 7s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 137m 38s Reason Tests Failed junit tests hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.server.namenode.TestReencryptionWithKMS   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestEncryptionZonesWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.server.namenode.TestReencryption   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886170/HDFS-12409.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3ccbb29aaf4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 738c2a9 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/21079/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21079/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21079/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 12/Sep/17 01:30 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 36s trunk passed +1 compile 1m 1s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 9s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 47s trunk passed       Patch Compile Tests +1 mvninstall 1m 5s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 110m 42s hadoop-hdfs in the patch failed. -1 asflicense 0m 22s The patch generated 1 ASF License warnings. 138m 54s Reason Tests Failed junit tests hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   hadoop.hdfs.TestFileAppendRestart   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestGetBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestModTime   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.server.namenode.TestReencryption   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 05a75bdea82b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66be6c8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21082/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/21082/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21082/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Andrew Wang added a comment - 12/Sep/17 18:54 I retriggered this since I just committed HDFS-12417 and the host had bind exceptions.

Hadoop QA added a comment - 12/Sep/17 20:57 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 16s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 57s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 0m 47s trunk passed       Patch Compile Tests +1 mvninstall 0m 59s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 0s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 47s the patch passed       Other Tests -1 unit 93m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 123m 16s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestLeaseRecoveryStriped Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886525/HDFS-12409.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d24ae41575f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f1d751b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21100/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21100/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21100/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Lei (Eddy) Xu added a comment - 14/Sep/17 00:15 Thanks Andrew Wang for the review. Committed to trunk.

Hudson added a comment - 14/Sep/17 00:32 SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12867/ ) HDFS-12409 . Add metrics of execution time of different stages in EC (lei: rev 73aed34dffa5e79f6f819137b69054c1dee2d4dd) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java