Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Attachments
Attachments
- HDFS-12637.1.patch
- 6 kB
- Takanobu Asanuma
Activity
Thanks tasanuma0829 for filing the jira and providing a patch.
I'm not familiar with EC details, but what's the difference with having a random non-default EC policy than parameterizing all policies for the base test (TestDistributedFileSystemWithECFile)? IIUC parameterizing will deterministically cover all policies, right?
Thanks for the comment, xiaochen.
Yes, that's right. But the random ec policy test is needed for a few reasons which are discussed in HDFS-7866 and HDFS-9962. I would like to quote Andrew's comment from HDFS-9962 that I agree.
We randomized the policy tests to reduce the runtime of the test suite, under the belief that there wasn't much incremental benefit from running all of them every time.
In summary, I'm working on this task following below:
- Default policy (RS-6-3) always gets tested.
- Create a random ec policy test if the test is long-running (e.g. using minicluster many times).
- If the test is short-running, all policies test with parametrizing would be better.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 21s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 14m 7s | trunk passed |
+1 | compile | 0m 53s | trunk passed |
+1 | checkstyle | 0m 39s | trunk passed |
+1 | mvnsite | 0m 58s | trunk passed |
+1 | shadedclient | 10m 19s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 46s | trunk passed |
+1 | javadoc | 0m 47s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 51s | the patch passed |
+1 | compile | 0m 48s | the patch passed |
-1 | javac | 0m 48s | hadoop-hdfs-project_hadoop-hdfs generated 139 new + 255 unchanged - 139 fixed = 394 total (was 394) |
+1 | checkstyle | 0m 34s | the patch passed |
+1 | mvnsite | 0m 54s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 23s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 2s | the patch passed |
+1 | javadoc | 0m 49s | the patch passed |
Other Tests | |||
-1 | unit | 118m 21s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
164m 40s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:0de40f0 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12892030/HDFS-12637.1.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 9ee8857698cd 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f4fb669 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/21682/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21682/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21682/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21682/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks tasanuma0829 for the pointer and explanation. Makes sense to me. The test only took a few seconds, but I think the rule of using parameterized tests only when no minicluster is started sounds better.
Change LGTM, pre-commit failures looks irrelevant. Retriggered a new run.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 19s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 14m 31s | trunk passed |
+1 | compile | 0m 55s | trunk passed |
+1 | checkstyle | 0m 39s | trunk passed |
+1 | mvnsite | 1m 6s | trunk passed |
+1 | shadedclient | 11m 28s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 59s | trunk passed |
+1 | javadoc | 0m 49s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 56s | the patch passed |
+1 | compile | 0m 51s | the patch passed |
-1 | javac | 0m 51s | hadoop-hdfs-project_hadoop-hdfs generated 139 new + 255 unchanged - 139 fixed = 394 total (was 394) |
+1 | checkstyle | 0m 36s | the patch passed |
+1 | mvnsite | 0m 55s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 11s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 7s | the patch passed |
+1 | javadoc | 0m 52s | the patch passed |
Other Tests | |||
-1 | unit | 125m 10s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
173m 24s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
hadoop.hdfs.server.namenode.ha.TestBootstrapStandby | |
hadoop.hdfs.TestPersistBlocks | |
hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | |
hadoop.hdfs.server.datanode.TestDirectoryScanner |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:0de40f0 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12892030/HDFS-12637.1.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 4d8cb0823d06 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f4fb669 |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/21689/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21689/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21689/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21689/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks for the review, xiaochen. The new failed tests don't seem to be related to the patch.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 13s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 20m 44s | trunk passed |
+1 | compile | 1m 24s | trunk passed |
+1 | checkstyle | 0m 53s | trunk passed |
+1 | mvnsite | 1m 24s | trunk passed |
+1 | shadedclient | 13m 14s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 36s | trunk passed |
+1 | javadoc | 1m 12s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 32s | the patch passed |
+1 | compile | 1m 27s | the patch passed |
+1 | javac | 1m 27s | the patch passed |
+1 | checkstyle | 0m 55s | the patch passed |
+1 | mvnsite | 1m 31s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 13m 23s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 49s | the patch passed |
+1 | javadoc | 1m 9s | the patch passed |
Other Tests | |||
-1 | unit | 124m 51s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
189m 45s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
hadoop.hdfs.TestPersistBlocks | |
hadoop.hdfs.qjournal.client.TestQuorumJournalManager | |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | |
hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
hadoop.hdfs.server.datanode.TestDirectoryScanner | |
hadoop.hdfs.TestEncryptionZones |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:0de40f0 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12892030/HDFS-12637.1.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 15f9ee55741d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 87ea1df |
Default Java | 1.8.0_144 |
findbugs | v3.1.0-RC1 |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21707/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21707/testReport/ |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21707/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Committed to trunk and branch-3.0.
Thanks for the contribution tasanuma0829!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13089 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13089/)
HDFS-12637. Extend TestDistributedFileSystemWithECFile with a random EC (xiao: rev 7bd700941d0a423d5232331fa19eab5fdab6a6fb)
- (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystemWithECFileWithRandomECPolicy.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystemWithECFile.java
Uploaded the 1st patch. The new test class with a random ec policy extends TestDistributedFileSystemWithECFile with little change. I checked all EC policies pass the tests in my local computer.