Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs, shell, test
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I did not find unit test in trunk code for following cases:

      • HDFS command dfs -tail
      • HDFS command dfs -stat

      I think it still merits to have one though the commands have served us for years.

      1. HDFS-10892.000.patch
        9 kB
        Mingliang Liu
      2. HDFS-10892.001.patch
        11 kB
        Mingliang Liu
      3. HDFS-10892.002.patch
        11 kB
        Mingliang Liu
      4. HDFS-10892.003.patch
        11 kB
        Mingliang Liu
      5. HDFS-10892.004.patch
        10 kB
        Mingliang Liu
      6. HDFS-10892.005.patch
        10 kB
        Mingliang Liu

        Issue Links

          Activity

          Hide
          liuml07 Mingliang Liu added a comment -

          The v0 patch added tests for 'dfs -tail' and 'dfs -stat'. Will probably add more to this single patch as the logic is very similar.

          Show
          liuml07 Mingliang Liu added a comment - The v0 patch added tests for 'dfs -tail' and 'dfs -stat'. Will probably add more to this single patch as the logic is very similar.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 55s trunk passed
          +1 compile 0m 53s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 58s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 59s the patch passed
          +1 javadoc 0m 58s the patch passed
          -1 unit 67m 13s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          88m 34s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend
            hadoop.hdfs.TestEncryptionZonesWithKMS
            hadoop.hdfs.TestDFSShell



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10892
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830136/HDFS-10892.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux def74f3617d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6eb700e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16852/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16852/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 55s trunk passed +1 compile 0m 53s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 58s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 67m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 88m 34s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestHAAppend   hadoop.hdfs.TestEncryptionZonesWithKMS   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830136/HDFS-10892.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux def74f3617d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6eb700e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16852/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16852/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16852/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 13s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179)
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 57s the patch passed
          +1 javadoc 0m 57s the patch passed
          -1 unit 60m 50s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          82m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSShell
            hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10892
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830377/HDFS-10892.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6b2f037bbd75 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3ae652f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16867/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16867/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 13s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 0m 57s the patch passed -1 unit 60m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 82m 27s Reason Tests Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830377/HDFS-10892.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b2f037bbd75 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ae652f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16867/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16867/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16867/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 3s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 57m 59s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          77m 6s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDecommissionWithStriped
            hadoop.hdfs.TestDFSShell



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10892
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830424/HDFS-10892.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 27b7050f80ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 059058f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16872/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16872/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 179 unchanged - 0 fixed = 181 total (was 179) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 57m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 77m 6s Reason Tests Failed junit tests hadoop.hdfs.TestDecommissionWithStriped   hadoop.hdfs.TestDFSShell Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830424/HDFS-10892.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 27b7050f80ea 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 059058f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16872/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16872/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16872/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment - - edited

          The hadoop.hdfs.TestDFSShell#testUtf8Encoding was not able to pass in Jenkins. The stack trace is like:

          Error Message

          Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
          Stacktrace

          java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
          at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
          at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
          at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
          at java.io.File.toPath(File.java:2234)
          at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getLastAccessTime(RawLocalFileSystem.java:662)
          at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.<init>(RawLocalFileSystem.java:673)
          at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:643)
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:871)
          at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:635)
          at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
          at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:360)
          at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2093)
          at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2061)
          at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2026)
          at org.apache.hadoop.hdfs.TestDFSShell.testUtf8Encoding(TestDFSShell.java:3885)

          I was able to pass the test locally without any problem. Perhaps it has something to with the LANG environment variable according to discussions on the Internet. I can confirm that my local test machine is using following settings.

          $ locale
          LANG="en_US.UTF-8"
          LC_COLLATE="en_US.UTF-8"
          LC_CTYPE="en_US.UTF-8"
          LC_MESSAGES="en_US.UTF-8"
          LC_MONETARY="en_US.UTF-8"
          LC_NUMERIC="en_US.UTF-8"
          LC_TIME="en_US.UTF-8"
          LC_ALL=
          

          I also think Yetus is setting the local correctly. Allen Wittenauer Do you have any idea about this? Thanks.

          As this test case is not tightly related to the commands "-tail" or "-stat", I remove this test case in the latest patch. If anyone suggests a working approach, I'd like to submit separate JIRA for tracking this test. Otherwise, we test it elsewhere (say, nightly system tests). Just for what it's worth, the test code is:

            /**
             * Test that the file name and content can have UTF-8 chars.
             */
            @Test (timeout = 30000)
            public void testUtf8Encoding() throws Exception {
              final int blockSize = 1024;
              final Configuration conf = new HdfsConfiguration();
              conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize);
          
              try (MiniDFSCluster cluster =
                       new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) {
                cluster.waitActive();
                final DistributedFileSystem dfs = cluster.getFileSystem();
                final Path workDir= new Path("/testUtf8Encoding");
                dfs.mkdirs(workDir);
          
                System.setProperty("sun.jnu.encoding", "UTF-8");
                System.setProperty("file.encoding", "UTF-8");
                final String chineseStr = "哈杜普.txt";
                final File testFile = new File(TEST_ROOT_DIR, chineseStr);
                // create a local file; its content contains the Chinese file name
                createLocalFile(testFile);
                dfs.copyFromLocalFile(new Path(testFile.getPath()), workDir);
                assertTrue(dfs.exists(new Path(workDir, testFile.getName())));
          
                final ByteArrayOutputStream out = new ByteArrayOutputStream();
                System.setOut(new PrintStream(out));
          
                final String argv[] = new String[]{
                    "-cat", workDir + "/" + testFile.getName()};
                final int ret = ToolRunner.run(new FsShell(conf), argv);
                assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret,
                    0, ret);
                assertTrue("Unexpected -cat output: " + out,
                    out.toString().contains(chineseStr));
              }
            }
          
          Show
          liuml07 Mingliang Liu added a comment - - edited The hadoop.hdfs.TestDFSShell#testUtf8Encoding was not able to pass in Jenkins. The stack trace is like: Error Message Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt Stacktrace java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt at sun.nio.fs.UnixPath.encode(UnixPath.java:147) at sun.nio.fs.UnixPath.<init>(UnixPath.java:71) at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281) at java.io.File.toPath(File.java:2234) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getLastAccessTime(RawLocalFileSystem.java:662) at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.<init>(RawLocalFileSystem.java:673) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:643) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:871) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:635) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:360) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2093) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2061) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2026) at org.apache.hadoop.hdfs.TestDFSShell.testUtf8Encoding(TestDFSShell.java:3885) I was able to pass the test locally without any problem. Perhaps it has something to with the LANG environment variable according to discussions on the Internet. I can confirm that my local test machine is using following settings. $ locale LANG= "en_US.UTF-8" LC_COLLATE= "en_US.UTF-8" LC_CTYPE= "en_US.UTF-8" LC_MESSAGES= "en_US.UTF-8" LC_MONETARY= "en_US.UTF-8" LC_NUMERIC= "en_US.UTF-8" LC_TIME= "en_US.UTF-8" LC_ALL= I also think Yetus is setting the local correctly. Allen Wittenauer Do you have any idea about this? Thanks. As this test case is not tightly related to the commands "-tail" or "-stat", I remove this test case in the latest patch. If anyone suggests a working approach, I'd like to submit separate JIRA for tracking this test. Otherwise, we test it elsewhere (say, nightly system tests). Just for what it's worth, the test code is: /** * Test that the file name and content can have UTF-8 chars. */ @Test (timeout = 30000) public void testUtf8Encoding() throws Exception { final int blockSize = 1024; final Configuration conf = new HdfsConfiguration(); conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize); try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) { cluster.waitActive(); final DistributedFileSystem dfs = cluster.getFileSystem(); final Path workDir= new Path( "/testUtf8Encoding" ); dfs.mkdirs(workDir); System .setProperty( "sun.jnu.encoding" , "UTF-8" ); System .setProperty( "file.encoding" , "UTF-8" ); final String chineseStr = "哈杜普.txt" ; final File testFile = new File(TEST_ROOT_DIR, chineseStr); // create a local file; its content contains the Chinese file name createLocalFile(testFile); dfs.copyFromLocalFile( new Path(testFile.getPath()), workDir); assertTrue(dfs.exists( new Path(workDir, testFile.getName()))); final ByteArrayOutputStream out = new ByteArrayOutputStream(); System .setOut( new PrintStream(out)); final String argv[] = new String []{ "-cat" , workDir + "/" + testFile.getName()}; final int ret = ToolRunner.run( new FsShell(conf), argv); assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret, 0, ret); assertTrue( "Unexpected -cat output: " + out, out.toString().contains(chineseStr)); } }
          Hide
          aw Allen Wittenauer added a comment - - edited

          The Dockerfile that ships with Hadoop does not force a UTF-8 locale. So it will inherit whatever locale was running at the UNIX level. (This, BTW, is intentional.) Apache Yetus will only force the locale to be a UTF-8 variant if shellcheck is involved.

          I'd recommend that if the test needs UTF-8, then the test should set the locale it needs and then restore. If setting the UTF-8 locale fails, then the test should be skipped. There's no guarantee that a user running the test is even UTF-8 capable.

          Show
          aw Allen Wittenauer added a comment - - edited The Dockerfile that ships with Hadoop does not force a UTF-8 locale. So it will inherit whatever locale was running at the UNIX level. (This, BTW, is intentional.) Apache Yetus will only force the locale to be a UTF-8 variant if shellcheck is involved. I'd recommend that if the test needs UTF-8, then the test should set the locale it needs and then restore. If setting the UTF-8 locale fails, then the test should be skipped. There's no guarantee that a user running the test is even UTF-8 capable.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 7s trunk passed
          +1 compile 0m 52s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 57s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 51s the patch passed
          +1 javac 0m 51s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179)
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 1m 0s the patch passed
          -1 unit 68m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          90m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.server.datanode.TestDataNodeUUID
            hadoop.hdfs.server.namenode.ha.TestHAAppend



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10892
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830586/HDFS-10892.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7e29a1340365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2acfb1e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16889/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16889/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 7s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 57s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 179 unchanged - 0 fixed = 180 total (was 179) +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 1m 0s the patch passed -1 unit 68m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 90m 11s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDataNodeUUID   hadoop.hdfs.server.namenode.ha.TestHAAppend Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10892 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830586/HDFS-10892.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7e29a1340365 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2acfb1e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16889/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16889/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16889/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          Thank you Allen Wittenauer for your explanation. That is very helpful to understand why the locale was not enforced to UTF-8 in test.

          However, I did not find easy way to set the LANG=UTF-8 encoding in Java (kindly let me know if there is). The Locale.setDefault() seems not working here either. Per-offline discussion with Jitendra Nath Pandey, we decide not to unit test UTF-8 by now. Mixing locale and file content encoding is better tested in end-to-end system tests.

          Show
          liuml07 Mingliang Liu added a comment - Thank you Allen Wittenauer for your explanation. That is very helpful to understand why the locale was not enforced to UTF-8 in test. However, I did not find easy way to set the LANG=UTF-8 encoding in Java (kindly let me know if there is). The Locale.setDefault() seems not working here either. Per-offline discussion with Jitendra Nath Pandey , we decide not to unit test UTF-8 by now. Mixing locale and file content encoding is better tested in end-to-end system tests.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          I think it makes sense to track utf8 tests outside this jira. These tests look good to me. +1

          Show
          jnp Jitendra Nath Pandey added a comment - I think it makes sense to track utf8 tests outside this jira. These tests look good to me. +1
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk through branch-2.8. Thanks Jitendra Nath Pandey for review.

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk through branch-2.8 . Thanks Jitendra Nath Pandey for review.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10509/)
          HDFS-10892. Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. (liuml07: rev 84c626407925e03ee2ef11faba9324d5c55b8e93)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10509 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10509/ ) HDFS-10892 . Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. (liuml07: rev 84c626407925e03ee2ef11faba9324d5c55b8e93) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development