Mingliang Liu added a comment - 23/Sep/16 22:53 The v0 patch added tests for 'dfs -tail' and 'dfs -stat'. Will probably add more to this single patch as the logic is very similar.

Hadoop QA added a comment - 24/Sep/16 02:14 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 53s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 67m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 88m 34s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830136/HDFS-10892.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux def74f3617d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6eb700e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16852/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16852/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 26/Sep/16 21:50 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 13s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 60m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 82m 27s Reason Tests Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830377/HDFS-10892.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b2f037bbd75 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ae652f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16867/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16867/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Hadoop QA added a comment - 27/Sep/16 02:27 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 57m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 77m 6s Reason Tests Failed junit tests hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830424/HDFS-10892.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27b7050f80ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 059058f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16872/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16872/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Mingliang Liu added a comment - 27/Sep/16 22:09 - edited The hadoop.hdfs.TestDFSShell#testUtf8Encoding was not able to pass in Jenkins. The stack trace is like: Error Message Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt Stacktrace java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt at sun.nio.fs.UnixPath.encode(UnixPath.java:147) at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) at java.io.File.toPath(File.java:2234) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getLastAccessTime(RawLocalFileSystem.java:662) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.<init>(RawLocalFileSystem.java:673) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:643) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:871) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:635) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:360) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2093) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2061) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2026) at org.apache.hadoop.hdfs.TestDFSShell.testUtf8Encoding(TestDFSShell.java:3885) I was able to pass the test locally without any problem. Perhaps it has something to with the LANG environment variable according to discussions on the Internet. I can confirm that my local test machine is using following settings. $ locale LANG= "en_US.UTF-8" LC_COLLATE= "en_US.UTF-8" LC_CTYPE= "en_US.UTF-8" LC_MESSAGES= "en_US.UTF-8" LC_MONETARY= "en_US.UTF-8" LC_NUMERIC= "en_US.UTF-8" LC_TIME= "en_US.UTF-8" LC_ALL= I also think Yetus is setting the local correctly. Allen Wittenauer Do you have any idea about this? Thanks. As this test case is not tightly related to the commands "-tail" or "-stat", I remove this test case in the latest patch. If anyone suggests a working approach, I'd like to submit separate JIRA for tracking this test. Otherwise, we test it elsewhere (say, nightly system tests). Just for what it's worth, the test code is: /** * Test that the file name and content can have UTF-8 chars. */ @Test (timeout = 30000) public void testUtf8Encoding() throws Exception { final int blockSize = 1024; final Configuration conf = new HdfsConfiguration(); conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize); try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) { cluster.waitActive(); final DistributedFileSystem dfs = cluster.getFileSystem(); final Path workDir= new Path( "/testUtf8Encoding" ); dfs.mkdirs(workDir); System .setProperty( "sun.jnu.encoding" , "UTF-8" ); System .setProperty( "file.encoding" , "UTF-8" ); final String chineseStr = "哈杜普.txt" ; final File testFile = new File(TEST_ROOT_DIR, chineseStr); // create a local file; its content contains the Chinese file name createLocalFile(testFile); dfs.copyFromLocalFile( new Path(testFile.getPath()), workDir); assertTrue(dfs.exists( new Path(workDir, testFile.getName()))); final ByteArrayOutputStream out = new ByteArrayOutputStream(); System .setOut( new PrintStream(out)); final String argv[] = new String []{ "-cat" , workDir + "/" + testFile.getName()}; final int ret = ToolRunner.run( new FsShell(conf), argv); assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret, 0, ret); assertTrue( "Unexpected -cat output: " + out, out.toString().contains(chineseStr)); } }

Allen Wittenauer added a comment - 27/Sep/16 23:22 - edited The Dockerfile that ships with Hadoop does not force a UTF-8 locale. So it will inherit whatever locale was running at the UNIX level. (This, BTW, is intentional.) Apache Yetus will only force the locale to be a UTF-8 variant if shellcheck is involved. I'd recommend that if the test needs UTF-8, then the test should set the locale it needs and then restore. If setting the UTF-8 locale fails, then the test should be skipped. There's no guarantee that a user running the test is even UTF-8 capable.

Hadoop QA added a comment - 28/Sep/16 00:40 -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 7s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 68m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 90m 11s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830586/HDFS-10892.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e29a1340365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2acfb1e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16889/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16889/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

Mingliang Liu added a comment - 28/Sep/16 05:14 Thank you Allen Wittenauer for your explanation. That is very helpful to understand why the locale was not enforced to UTF-8 in test. However, I did not find easy way to set the LANG=UTF-8 encoding in Java (kindly let me know if there is). The Locale.setDefault() seems not working here either. Per-offline discussion with Jitendra Nath Pandey , we decide not to unit test UTF-8 by now. Mixing locale and file content encoding is better tested in end-to-end system tests.

Jitendra Nath Pandey added a comment - 28/Sep/16 20:25 I think it makes sense to track utf8 tests outside this jira. These tests look good to me. +1

Mingliang Liu added a comment - 28/Sep/16 23:12 Committed to trunk through branch-2.8 . Thanks Jitendra Nath Pandey for review.

Hudson added a comment - 28/Sep/16 23:17 SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10509/ ) HDFS-10892 . Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. (liuml07: rev 84c626407925e03ee2ef11faba9324d5c55b8e93) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java