Wei Zhou added a comment - 15/Aug/17 10:10 Initial patch.

Hadoop QA added a comment - 15/Aug/17 12:47 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 18m 13s trunk passed +1 compile 1m 53s trunk passed +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 53s trunk passed +1 findbugs 4m 0s trunk passed +1 javadoc 1m 21s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 1m 49s the patch passed +1 javac 1m 49s the patch passed +1 checkstyle 0m 51s the patch passed +1 mvnsite 1m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 7s the patch passed       Other Tests +1 unit 1m 29s hadoop-hdfs-client in the patch passed. -1 unit 103m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 147m 2s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881906/HDFS-12303.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e45c982d5cd5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e43c28 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20700/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20700/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Wei Zhou added a comment - 16/Aug/17 05:57 The junit test failures reported can not be reproduced in my local environment. Resubmit the patch to trigger another round Hadoop QA check. Thanks!

Andrew Wang added a comment - 17/Aug/17 16:59 Hi Wei, thanks for working on this! A single review comment: I notice there's still an old reference to XOR-64k in TestDistCpUtils Toggling the JIRA status doesn't retrigger precommit, you'd need to re-upload the same patch. I ran a few of the failed tests and saw that they succeeded. Hoping it's just a flake.

Wei Zhou added a comment - 21/Aug/17 02:41 Thanks Andrew Wang for reviewing the patch and clarifying the JIRA details. The old references were removed from TestDistCpUtils and patch rebased. Thanks!

Hadoop QA added a comment - 21/Aug/17 05:38 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 1m 30s Maven dependency ordering for branch +1 mvninstall 13m 52s trunk passed +1 compile 14m 12s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 2m 9s trunk passed +1 findbugs 3m 47s trunk passed +1 javadoc 1m 33s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 10m 41s the patch passed +1 javac 10m 41s the patch passed +1 checkstyle 2m 4s the patch passed +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 4m 11s the patch passed +1 javadoc 1m 35s the patch passed       Other Tests +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 87m 17s hadoop-hdfs in the patch failed. +1 unit 13m 54s hadoop-distcp in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 166m 51s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882807/HDFS-12303.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 412b28b69b4b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7a82d7b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20772/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20772/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Kai Zheng added a comment - 21/Aug/17 05:49 Just took a look at the failed unit tests, finding most of them reported the following line: at org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780) pointing to this line of code Preconditions.checkState(ackedBytes <= sentBytes); which is being handled under HDFS-11882 .

Wei Zhou added a comment - 23/Aug/17 06:19 Thanks Kai Zheng for review the patch. Those cases that has connection with updatePipeline are all passed in my testing, so it should have no connection with issue. This patch fixes an issue in TestBlockManager and mainly for another round of checking. Thanks!

Hadoop QA added a comment - 23/Aug/17 09:14 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 1m 47s Maven dependency ordering for branch +1 mvninstall 15m 26s trunk passed +1 compile 14m 20s trunk passed +1 checkstyle 2m 7s trunk passed +1 mvnsite 2m 8s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 36s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 10m 45s the patch passed +1 javac 10m 45s the patch passed +1 checkstyle 2m 8s the patch passed +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 34s the patch passed       Other Tests +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 90m 44s hadoop-hdfs in the patch failed. +1 unit 14m 26s hadoop-distcp in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 173m 6s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883267/HDFS-12303.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 486a492d3218 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4249172 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20817/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20817/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Andrew Wang added a comment - 25/Aug/17 21:13 +1 LGTM, will check this in.

Andrew Wang added a comment - 25/Aug/17 21:14 Committed to trunk. Thanks Wei for the nice contribution and Kai for reviewing!

Hudson added a comment - 25/Aug/17 21:52 SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12246/ ) HDFS-12303 . Change default EC cell size to 1MB for better performance. (wang: rev f29a0fc288a625522ba910e61b63fd5f10418b3d) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/TestErasureCodingCLI.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/test_ec_policies.xml