Details
-
Type:
New Feature
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 3.0.0-alpha2
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Reviewed
Description
HDFS erasure code policy allows user to store directory and file to predefined erasure code policies. Currently only 3x replication is supported in TestDFSIO implementation. This is going to add an new option to enable tests of files with erasure code policy enabled.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
1. improve the patch, only io_data directory is mattered when apply EC policy
2. fix checkstyle reported issue
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 11s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 22s | trunk passed |
| +1 | compile | 0m 23s | trunk passed |
| +1 | checkstyle | 0m 17s | trunk passed |
| +1 | mvnsite | 0m 30s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 0m 24s | trunk passed |
| +1 | javadoc | 0m 12s | trunk passed |
| +1 | mvninstall | 0m 21s | the patch passed |
| +1 | compile | 0m 21s | the patch passed |
| +1 | javac | 0m 21s | the patch passed |
| -1 | checkstyle | 0m 14s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 5 new + 45 unchanged - 5 fixed = 50 total (was 50) |
| +1 | mvnsite | 0m 25s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 30s | the patch passed |
| +1 | javadoc | 0m 10s | the patch passed |
| +1 | unit | 123m 7s | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| 136m 0s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827513/MAPREDUCE-6774-v2.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 3b6c36917bf8 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 63f5948 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6708/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 45s | trunk passed |
| +1 | compile | 0m 24s | trunk passed |
| +1 | checkstyle | 0m 16s | trunk passed |
| +1 | mvnsite | 0m 30s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 0m 25s | trunk passed |
| +1 | javadoc | 0m 13s | trunk passed |
| +1 | mvninstall | 0m 22s | the patch passed |
| +1 | compile | 0m 21s | the patch passed |
| +1 | javac | 0m 21s | the patch passed |
| +1 | checkstyle | 0m 13s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) |
| +1 | mvnsite | 0m 26s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 28s | the patch passed |
| +1 | javadoc | 0m 10s | the patch passed |
| -1 | unit | 135m 32s | hadoop-mapreduce-client-jobclient in the patch failed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 147m 51s |
| Reason | Tests |
|---|---|
| Timed out junit tests | org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827546/MAPREDUCE-6774-v3.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux cf3bca10b7c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 63f5948 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
| unit test logs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6709/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Thanks SammiChen for the work!
It looks good overall. Suggestions:
boolean isStoragePolicyValid(String storagePolicy, FileSystem fs) boolean isErasureCodePolicyValid(String erasureCodePolicyName, FileSystem fs, TestType testType) throws IOException
would be good to:
void checkStoragePolicy(String storagePolicy, FileSystem fs) throws Exception void checkErasureCodePolicy(String erasureCodePolicyName, FileSystem fs, TestType testType) throws Exception
So given above, the following can then be:
+ if (storagePolicy != null) { + if (!isStoragePolicyValid(storagePolicy, fs)) { + return -1; + }
=>
+ if (storagePolicy != null) { + checkStoragePolicy(storagePolicy, fs); + }
Thanks Kai, for take time review the patch and give very good suggestion!
The name 'checkStoragePolicy' sounds better than 'isStoragePolicyValid'. So does the name 'checkErasureCodePolicy'. I uploaded Patch v4 which contains the improvement. One the other hand, I would suggest keep the return value boolean of each check function. Because the main function depends on the return value, not the exception, to handle the invalid parameter case.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 57s | trunk passed |
| +1 | compile | 0m 23s | trunk passed |
| +1 | checkstyle | 0m 17s | trunk passed |
| +1 | mvnsite | 0m 30s | trunk passed |
| +1 | mvneclipse | 0m 16s | trunk passed |
| +1 | findbugs | 0m 23s | trunk passed |
| +1 | javadoc | 0m 13s | trunk passed |
| +1 | mvninstall | 0m 22s | the patch passed |
| +1 | compile | 0m 20s | the patch passed |
| +1 | javac | 0m 20s | the patch passed |
| +1 | checkstyle | 0m 14s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) |
| +1 | mvnsite | 0m 27s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 28s | the patch passed |
| +1 | javadoc | 0m 10s | the patch passed |
| +1 | unit | 117m 35s | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| 130m 9s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828201/MAPREDUCE-6774-v4.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux d30c2069690d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 729de3e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6716/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Thanks Sammi for the update! It looks good and only some minors now.
1. Could you define constants for: test.io.erasure.code.policy, test.io.block.storage.policy?
2. Would like to see minor refinements for checkErasureCodePolicy. For example, having some line breaks, avoiding the else.
+ private boolean checkErasureCodePolicy(String erasureCodePolicyName, + FileSystem fs, TestType testType) throws IOException { + Collection<ErasureCodingPolicy> list = + ((DistributedFileSystem) fs).getAllErasureCodingPolicies(); + boolean isValid = false; + int i = 0; + for (ErasureCodingPolicy ec : list) { + if (erasureCodePolicyName.equals(ec.getName())) { + isValid = true; + break; + } + } + if (!isValid) { + System.out.println("Invalid erasure code policy: " + + erasureCodePolicyName); + System.out.println("Current supported erasure code policy list: "); + for (ErasureCodingPolicy ec : list) { + System.out.println(ec.getName()); + } + return false; + } else { + if (testType == TestType.TEST_TYPE_APPEND || + testType == TestType.TEST_TYPE_TRUNCATE) { + System.out.println("So far append or truncate operation" + + " with erasureCodePolicy enabled is not supported"); + return false; + } + } + config.set("test.io.erasure.code.policy", erasureCodePolicyName); + LOG.info("erasureCodePolicy = " + erasureCodePolicyName); + return true; + }
Improved patch based on Kai's suggestion
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 41s | trunk passed |
| +1 | compile | 0m 23s | trunk passed |
| +1 | checkstyle | 0m 18s | trunk passed |
| +1 | mvnsite | 0m 29s | trunk passed |
| +1 | mvneclipse | 0m 16s | trunk passed |
| +1 | findbugs | 0m 24s | trunk passed |
| +1 | javadoc | 0m 13s | trunk passed |
| +1 | mvninstall | 0m 21s | the patch passed |
| +1 | compile | 0m 19s | the patch passed |
| +1 | javac | 0m 19s | the patch passed |
| -1 | checkstyle | 0m 14s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 2 new + 45 unchanged - 5 fixed = 47 total (was 50) |
| +1 | mvnsite | 0m 26s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 27s | the patch passed |
| +1 | javadoc | 0m 10s | the patch passed |
| +1 | unit | 126m 30s | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| 139m 35s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828406/MAPREDUCE-6774-v5.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 746e781eb2ed 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ea0c2b8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6717/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 26s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 8m 44s | trunk passed |
| +1 | compile | 0m 25s | trunk passed |
| +1 | checkstyle | 0m 18s | trunk passed |
| +1 | mvnsite | 0m 32s | trunk passed |
| +1 | mvneclipse | 0m 16s | trunk passed |
| +1 | findbugs | 0m 25s | trunk passed |
| +1 | javadoc | 0m 15s | trunk passed |
| +1 | mvninstall | 0m 22s | the patch passed |
| +1 | compile | 0m 21s | the patch passed |
| +1 | javac | 0m 21s | the patch passed |
| +1 | checkstyle | 0m 15s | hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 0 new + 45 unchanged - 5 fixed = 45 total (was 50) |
| +1 | mvnsite | 0m 28s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 29s | the patch passed |
| +1 | javadoc | 0m 11s | the patch passed |
| +1 | unit | 119m 42s | hadoop-mapreduce-client-jobclient in the patch passed. |
| +1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| 134m 30s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828433/MAPREDUCE-6774-v6.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux aaaf6aff956f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ea0c2b8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6718/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Thanks Sammi for the update! The latest patch LGTM and +1.
Committed to trunk. Thanks Sammi for the contribution!
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10452 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10452/)
MAPREDUCE-6774. Add support for HDFS erasure code policy to TestDFSIO. (kai.zheng: rev 501a77856d6b6edfb261547117e719da7a9cd221)
- (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
MAPREDUCE-6774This message was automatically generated.