Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
Currently the dfs -test command only supports
-d, -e, -f, -s, -z
options. It would be helpful if we add
-w, -r
to verify permission of r/w before actual read or write. This will help script programming.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I have a patch for this, this should be pretty straightforward, can someone help to review? Thanks a lot!
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 15s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 36s | trunk passed |
| +1 | compile | 6m 4s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 6m 54s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 1m 10s | trunk passed |
| +1 | mvnsite | 2m 5s | trunk passed |
| +1 | mvneclipse | 0m 29s | trunk passed |
| +1 | findbugs | 4m 3s | trunk passed |
| +1 | javadoc | 2m 9s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 3m 16s | trunk passed with JDK v1.7.0_95 |
| 0 | mvndep | 0m 15s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 39s | the patch passed |
| +1 | compile | 7m 47s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 7m 47s | the patch passed |
| +1 | compile | 8m 23s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 8m 23s | the patch passed |
| -1 | checkstyle | 1m 11s | root: patch generated 9 new + 32 unchanged - 0 fixed = 41 total (was 32) |
| +1 | mvnsite | 1m 58s | the patch passed |
| +1 | mvneclipse | 0m 29s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 4m 32s | the patch passed |
| +1 | javadoc | 2m 14s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 3m 6s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 8m 57s | hadoop-common in the patch passed with JDK v1.8.0_77. |
| -1 | unit | 73m 11s | hadoop-hdfs in the patch failed with JDK v1.8.0_77. |
| +1 | unit | 8m 6s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| -1 | unit | 67m 34s | hadoop-hdfs in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 26s | Patch does not generate ASF License warnings. |
| 224m 32s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.server.datanode.TestDataNodeLifeline | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding | |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.server.namenode.TestCheckpoint |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.server.namenode.ha.TestHAStateTransitions | |
| hadoop.hdfs.server.namenode.ha.TestQuotasWithHA | |
| hadoop.hdfs.TestHFlush | |
| hadoop.fs.loadGenerator.TestLoadGenerator | |
| hadoop.hdfs.server.namenode.TestNamenodeRetryCache | |
| hadoop.fs.contract.hdfs.TestHDFSContractMkdir | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
This message was automatically generated.
Upload v2 patch, resolve checkstyle issues and updated documents.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 8s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 15s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 38s | trunk passed |
| +1 | compile | 6m 9s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 6m 37s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 1m 3s | trunk passed |
| +1 | mvnsite | 1m 45s | trunk passed |
| +1 | mvneclipse | 0m 27s | trunk passed |
| +1 | findbugs | 3m 25s | trunk passed |
| +1 | javadoc | 2m 0s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 2m 47s | trunk passed with JDK v1.7.0_95 |
| 0 | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 29s | the patch passed |
| +1 | compile | 5m 59s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 5m 59s | the patch passed |
| +1 | compile | 6m 45s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 6m 45s | the patch passed |
| -1 | checkstyle | 1m 5s | root: patch generated 4 new + 8 unchanged - 24 fixed = 12 total (was 32) |
| +1 | mvnsite | 1m 58s | the patch passed |
| +1 | mvneclipse | 0m 31s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 4m 44s | the patch passed |
| +1 | javadoc | 2m 10s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 3m 5s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 8m 25s | hadoop-common in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 69m 40s | hadoop-hdfs in the patch failed with JDK v1.8.0_77. |
| +1 | unit | 8m 48s | hadoop-common in the patch passed with JDK v1.7.0_95. |
| -1 | unit | 62m 27s | hadoop-hdfs in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 26s | Patch does not generate ASF License warnings. |
| 210m 25s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.ha.TestZKFailoverController |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.server.namenode.TestEditLog | |
| hadoop.hdfs.TestRenameWhileOpen | |
| hadoop.hdfs.server.datanode.TestDataNodeLifeline | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestFileAppend | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 | |
| JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding | |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| hadoop.fs.shell.TestHdfsTextCommand | |
| hadoop.hdfs.TestErasureCodeBenchmarkThroughput | |
| hadoop.hdfs.TestReconstructStripedFile | |
| JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
This message was automatically generated.
Submit v3 patch, resolved a minor checkstyle issue.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 9s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 1m 1s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 31s | trunk passed |
| +1 | compile | 5m 37s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 6m 31s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 1m 4s | trunk passed |
| +1 | mvnsite | 1m 47s | trunk passed |
| +1 | mvneclipse | 0m 27s | trunk passed |
| +1 | findbugs | 3m 28s | trunk passed |
| +1 | javadoc | 1m 57s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 2m 46s | trunk passed with JDK v1.7.0_95 |
| 0 | mvndep | 0m 14s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 27s | the patch passed |
| +1 | compile | 5m 56s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 5m 56s | the patch passed |
| +1 | compile | 6m 53s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 6m 53s | the patch passed |
| +1 | checkstyle | 1m 3s | root: patch generated 0 new + 8 unchanged - 24 fixed = 8 total (was 32) |
| +1 | mvnsite | 1m 46s | the patch passed |
| +1 | mvneclipse | 0m 28s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 4m 7s | the patch passed |
| +1 | javadoc | 1m 56s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 2m 53s | the patch passed with JDK v1.7.0_95 |
| +1 | unit | 8m 9s | hadoop-common in the patch passed with JDK v1.8.0_77. |
| -1 | unit | 72m 45s | hadoop-hdfs in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 8m 58s | hadoop-common in the patch failed with JDK v1.7.0_95. |
| -1 | unit | 63m 21s | hadoop-hdfs in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 25s | Patch does not generate ASF License warnings. |
| 213m 5s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| hadoop.hdfs.TestRenameWhileOpen | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestFileAppend | |
| hadoop.hdfs.TestLocalDFS | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| hadoop.fs.permission.TestStickyBit | |
| JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding | |
| JDK v1.7.0_95 Failed junit tests | hadoop.ha.TestZKFailoverController |
| hadoop.fs.contract.hdfs.TestHDFSContractSeek | |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestLargeBlock | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile |
| org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 10s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 31s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 28s | trunk passed |
| +1 | compile | 6m 35s | trunk passed |
| +1 | checkstyle | 1m 21s | trunk passed |
| +1 | mvnsite | 1m 50s | trunk passed |
| +1 | mvneclipse | 0m 24s | trunk passed |
| +1 | findbugs | 3m 3s | trunk passed |
| +1 | javadoc | 2m 2s | trunk passed |
| 0 | mvndep | 0m 12s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 28s | the patch passed |
| +1 | compile | 6m 32s | the patch passed |
| +1 | javac | 6m 32s | the patch passed |
| -1 | checkstyle | 1m 22s | root: The patch generated 2 new + 186 unchanged - 25 fixed = 188 total (was 211) |
| +1 | mvnsite | 1m 48s | the patch passed |
| +1 | mvneclipse | 0m 25s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 3m 20s | the patch passed |
| +1 | javadoc | 1m 58s | the patch passed |
| -1 | unit | 7m 40s | hadoop-common in the patch failed. |
| +1 | unit | 58m 57s | hadoop-hdfs in the patch passed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 107m 24s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.net.TestDNS |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12805605/HADOOP-12943.004.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 17aa537908e5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6161d9b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/9550/artifact/patchprocess/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/9550/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit test logs | https://builds.apache.org/job/PreCommit-HADOOP-Build/9550/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9550/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9550/console |
| Powered by | Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The testDFSShell method already had over 150 lines and some nested blocks before my patch, I am not sure why it is listed as 2 new issues. It's hard to reduce lines of to <150 for this test case, I can fix the nested blocks, but that would modify more than just the issue this patch wants to fix. So probably not for now.
The failed test case is tracked by HADOOP-13101, not related to this patch.
Hello Akira Ajisaka appreciate if you can take a look at this one when you have time. Thanks a lot.
Mostly looks good to me. Thanks Weiwei Yang.
Test.java
} catch (AccessControlException e) {
return false;
} catch (FileNotFoundException e) {
return false;
}
Would you use multi-catch statement here? I'm +1 if that is addressed.
Thanks Akira Ajisaka I have addressed that in 005 patch, and submitted just now.
+1 pending Jenkins. Thanks Weiwei Yang.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 25s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| 0 | mvndep | 0m 12s | Maven dependency ordering for branch |
| +1 | mvninstall | 6m 47s | trunk passed |
| +1 | compile | 6m 58s | trunk passed |
| +1 | checkstyle | 1m 22s | trunk passed |
| +1 | mvnsite | 1m 55s | trunk passed |
| +1 | mvneclipse | 0m 25s | trunk passed |
| +1 | findbugs | 3m 8s | trunk passed |
| +1 | javadoc | 1m 51s | trunk passed |
| 0 | mvndep | 0m 11s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 29s | the patch passed |
| +1 | compile | 7m 7s | the patch passed |
| +1 | javac | 7m 7s | the patch passed |
| -1 | checkstyle | 1m 24s | root: The patch generated 2 new + 186 unchanged - 25 fixed = 188 total (was 211) |
| +1 | mvnsite | 2m 5s | the patch passed |
| +1 | mvneclipse | 0m 28s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 4m 3s | the patch passed |
| +1 | javadoc | 1m 57s | the patch passed |
| -1 | unit | 8m 36s | hadoop-common in the patch failed. |
| -1 | unit | 73m 9s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
| 124m 50s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| hadoop.hdfs.server.namenode.TestEditLog | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:e2f6409 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12810783/HADOOP-12943.005.patch |
| JIRA Issue | |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux c1cc34c4cfe8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 51d497f |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/9810/artifact/patchprocess/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/9810/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/9810/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9810/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9810/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1, the checkstyle warnings and test failures seem to be unrelated to the patch.
Committed this to trunk, branch-2, and branch-2.8. Thanks Weiwei Yang for the contribution.
Thanks Akira Ajisaka
SUCCESS: Integrated in Hadoop-trunk-Commit #9977 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9977/)
HADOOP-12943. Add -w -r options in dfs -test command. Contributed by (aajisaka: rev 09e82acaf9a6d7663bc51bbca0cdeca4b582b535)
- hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Test.java
Please let me know if this makes sense, I can help to contribute a patch if it does. Thanks.