Details
Description
I did not find unit test in trunk code for following cases:
- HDFS command dfs -tail
- HDFS command dfs -stat
I think it still merits to have one though the commands have served us for years.
Issue Links
- breaks
-
HDFS-10934
TestDFSShell.testStat fails intermittently
-
- Resolved
-
- is depended upon by
-
HADOOP-13522
Add %A and %a formats for fs -stat command to print permissions
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Committed to trunk through branch-2.8. Thanks Jitendra Nath Pandey for review.
I think it makes sense to track utf8 tests outside this jira. These tests look good to me. +1
Thank you Allen Wittenauer for your explanation. That is very helpful to understand why the locale was not enforced to UTF-8 in test.
However, I did not find easy way to set the LANG=UTF-8 encoding in Java (kindly let me know if there is). The Locale.setDefault() seems not working here either. Per-offline discussion with Jitendra Nath Pandey, we decide not to unit test UTF-8 by now. Mixing locale and file content encoding is better tested in end-to-end system tests.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 16s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 8m 7s | trunk passed |
| +1 | compile | 0m 52s | trunk passed |
| +1 | checkstyle | 0m 32s | trunk passed |
| +1 | mvnsite | 0m 59s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 50s | trunk passed |
| +1 | javadoc | 0m 57s | trunk passed |
| +1 | mvninstall | 0m 57s | the patch passed |
| +1 | compile | 0m 51s | the patch passed |
| +1 | javac | 0m 51s | the patch passed |
| -0 | checkstyle | 0m 29s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) |
| +1 | mvnsite | 0m 58s | the patch passed |
| +1 | mvneclipse | 0m 11s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 6s | the patch passed |
| +1 | javadoc | 1m 0s | the patch passed |
| -1 | unit | 68m 6s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 90m 11s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| hadoop.hdfs.server.datanode.TestDataNodeUUID | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830586/HDFS-10892.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 7e29a1340365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2acfb1e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16889/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16889/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The Dockerfile that ships with Hadoop does not force a UTF-8 locale. So it will inherit whatever locale was running at the UNIX level. (This, BTW, is intentional.) Apache Yetus will only force the locale to be a UTF-8 variant if shellcheck is involved.
I'd recommend that if the test needs UTF-8, then the test should set the locale it needs and then restore. If setting the UTF-8 locale fails, then the test should be skipped. There's no guarantee that a user running the test is even UTF-8 capable.
The hadoop.hdfs.TestDFSShell#testUtf8Encoding was not able to pass in Jenkins. The stack trace is like:
Error Message
Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
Stacktracejava.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
at java.io.File.toPath(File.java:2234)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getLastAccessTime(RawLocalFileSystem.java:662)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.<init>(RawLocalFileSystem.java:673)
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:643)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:871)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:635)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:360)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2093)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2061)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2026)
at org.apache.hadoop.hdfs.TestDFSShell.testUtf8Encoding(TestDFSShell.java:3885)
I was able to pass the test locally without any problem. Perhaps it has something to with the LANG environment variable according to discussions on the Internet. I can confirm that my local test machine is using following settings.
$ locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL=
I also think Yetus is setting the local correctly. Allen Wittenauer Do you have any idea about this? Thanks.
As this test case is not tightly related to the commands "-tail" or "-stat", I remove this test case in the latest patch. If anyone suggests a working approach, I'd like to submit separate JIRA for tracking this test. Otherwise, we test it elsewhere (say, nightly system tests). Just for what it's worth, the test code is:
/** * Test that the file name and content can have UTF-8 chars. */ @Test (timeout = 30000) public void testUtf8Encoding() throws Exception { final int blockSize = 1024; final Configuration conf = new HdfsConfiguration(); conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize); try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) { cluster.waitActive(); final DistributedFileSystem dfs = cluster.getFileSystem(); final Path workDir= new Path("/testUtf8Encoding"); dfs.mkdirs(workDir); System.setProperty("sun.jnu.encoding", "UTF-8"); System.setProperty("file.encoding", "UTF-8"); final String chineseStr = "哈杜普.txt"; final File testFile = new File(TEST_ROOT_DIR, chineseStr); // create a local file; its content contains the Chinese file name createLocalFile(testFile); dfs.copyFromLocalFile(new Path(testFile.getPath()), workDir); assertTrue(dfs.exists(new Path(workDir, testFile.getName()))); final ByteArrayOutputStream out = new ByteArrayOutputStream(); System.setOut(new PrintStream(out)); final String argv[] = new String[]{ "-cat", workDir + "/" + testFile.getName()}; final int ret = ToolRunner.run(new FsShell(conf), argv); assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret, 0, ret); assertTrue("Unexpected -cat output: " + out, out.toString().contains(chineseStr)); } }
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 3s | trunk passed |
| +1 | compile | 0m 46s | trunk passed |
| +1 | checkstyle | 0m 27s | trunk passed |
| +1 | mvnsite | 0m 51s | trunk passed |
| +1 | mvneclipse | 0m 12s | trunk passed |
| +1 | findbugs | 1m 39s | trunk passed |
| +1 | javadoc | 0m 54s | trunk passed |
| +1 | mvninstall | 0m 46s | the patch passed |
| +1 | compile | 0m 41s | the patch passed |
| +1 | javac | 0m 41s | the patch passed |
| -0 | checkstyle | 0m 24s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) |
| +1 | mvnsite | 0m 48s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 46s | the patch passed |
| +1 | javadoc | 0m 52s | the patch passed |
| -1 | unit | 57m 59s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 77m 6s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestDecommissionWithStriped |
| hadoop.hdfs.TestDFSShell |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830424/HDFS-10892.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 27b7050f80ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 059058f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16872/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16872/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 23s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 8m 13s | trunk passed |
| +1 | compile | 0m 49s | trunk passed |
| +1 | checkstyle | 0m 30s | trunk passed |
| +1 | mvnsite | 0m 56s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 1m 50s | trunk passed |
| +1 | javadoc | 1m 1s | trunk passed |
| +1 | mvninstall | 0m 52s | the patch passed |
| +1 | compile | 0m 45s | the patch passed |
| +1 | javac | 0m 45s | the patch passed |
| -0 | checkstyle | 0m 25s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) |
| +1 | mvnsite | 0m 54s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 57s | the patch passed |
| +1 | javadoc | 0m 57s | the patch passed |
| -1 | unit | 60m 50s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| 82m 27s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestDFSShell |
| hadoop.hdfs.TestLeaseRecovery2 |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830377/HDFS-10892.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 6b2f037bbd75 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ae652f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16867/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16867/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 55s | trunk passed |
| +1 | compile | 0m 53s | trunk passed |
| +1 | checkstyle | 0m 30s | trunk passed |
| +1 | mvnsite | 0m 59s | trunk passed |
| +1 | mvneclipse | 0m 12s | trunk passed |
| +1 | findbugs | 1m 50s | trunk passed |
| +1 | javadoc | 0m 58s | trunk passed |
| +1 | mvninstall | 0m 53s | the patch passed |
| +1 | compile | 0m 49s | the patch passed |
| +1 | javac | 0m 49s | the patch passed |
| -0 | checkstyle | 0m 28s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) |
| +1 | mvnsite | 1m 0s | the patch passed |
| +1 | mvneclipse | 0m 11s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 59s | the patch passed |
| +1 | javadoc | 0m 58s | the patch passed |
| -1 | unit | 67m 13s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 88m 34s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| hadoop.hdfs.TestEncryptionZonesWithKMS | |
| hadoop.hdfs.TestDFSShell |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12830136/HDFS-10892.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux def74f3617d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6eb700e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16852/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16852/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The v0 patch added tests for 'dfs -tail' and 'dfs -stat'. Will probably add more to this single patch as the logic is very similar.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10509/)
HDFS-10892. Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. (liuml07: rev 84c626407925e03ee2ef11faba9324d5c55b8e93)