Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Environment:

      OS: Windows Server 2012
      JDK: Oracle 1.7.0_79

    • Hadoop Flags:
      Reviewed

      Description

      TestTeraSort testcase fails on Windows.

      The test case uses the build directory as test working directory.
      Under Windows the build directory starts with a drive definition ( "C:" ),
      which is interpreted as (an invalid) URI scheme.

      The fix is trivial: Add URI scheme to the beginning of the working directory.

      Error message:

      Running org.apache.hadoop.examples.terasort.TestTeraSort
      Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.647 sec <<< FAILURE! - in org.apache.hadoop.examples.terasort.TestTeraSort
      testTeraSort(org.apache.hadoop.examples.terasort.TestTeraSort)  Time elapsed: 3.359 sec  <<< ERROR!
      java.io.IOException: No FileSystem for scheme: C
              at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2787)
              at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2798)
              at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
              at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2837)
              at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2819)
              at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:381)
              at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:223)
              at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
              at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
              at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:179)
              at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:98)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:193)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
              at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
              at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:331)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.examples.terasort.TestTeraSort.runTeraSort(TestTeraSort.java:75)
              at org.apache.hadoop.examples.terasort.TestTeraSort.testTeraSort(TestTeraSort.java:101)
      
      
      Results :
      
      Tests in error:
        TestTeraSort.testTeraSort:101->runTeraSort:75 ╗ IO No FileSystem for scheme: C
      
      Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
      
      1. MAPREDUCE-6672.01.patch
        1 kB
        Tibor Kiss
      2. MAPREDUCE-6672.02.patch
        1 kB
        Tibor Kiss

        Issue Links

          Activity

          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Attached patch file which fixes the TestTeraSort UT failure.
          Note that the URI scheme is different between UNIX-like (Linux & OS X) and Windows operating systems.

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Attached patch file which fixes the TestTeraSort UT failure. Note that the URI scheme is different between UNIX-like (Linux & OS X) and Windows operating systems.
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          The patch has been verified on Windows Server 2012, OS X 10.11 & CentOS 7.2.

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - The patch has been verified on Windows Server 2012, OS X 10.11 & CentOS 7.2.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 36s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_77
          +1 compile 0m 15s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 34s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_77
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_77
          +1 javac 0m 10s the patch passed
          +1 compile 0m 13s the patch passed with JDK v1.7.0_95
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 24s hadoop-mapreduce-examples in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 28s hadoop-mapreduce-examples in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          13m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797907/MAPREDUCE-6672.01.patch
          JIRA Issue MAPREDUCE-6672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6564bebc6ab8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ff95fd5
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6425/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project/hadoop-mapreduce-examples
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6425/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 36s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_77 +1 compile 0m 15s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 34s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 16s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_77 +1 javac 0m 10s the patch passed +1 compile 0m 13s the patch passed with JDK v1.7.0_95 +1 javac 0m 13s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 24s hadoop-mapreduce-examples in the patch passed with JDK v1.8.0_77. +1 unit 0m 28s hadoop-mapreduce-examples in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 13m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797907/MAPREDUCE-6672.01.patch JIRA Issue MAPREDUCE-6672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6564bebc6ab8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ff95fd5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6425/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project/hadoop-mapreduce-examples Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6425/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks for the patch, Tibor Kiss. The fix may be right... however, hard code on the path with different OS prefix may not be the best choice.
          I think it is better to use

            private static final Path TEST_DIR = new Path(System.getProperty("test.build.data", "/tmp"), "terasort");
          

          That should work on Windows given we have many other tests follow the same pattern.
          CC Chris Nauroth who is Windows expert.

          Show
          djp Junping Du added a comment - Thanks for the patch, Tibor Kiss . The fix may be right... however, hard code on the path with different OS prefix may not be the best choice. I think it is better to use private static final Path TEST_DIR = new Path(System.getProperty("test.build.data", "/tmp"), "terasort"); That should work on Windows given we have many other tests follow the same pattern. CC Chris Nauroth who is Windows expert.
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Thanks Junping Du for the proposed enhancement. I agree that it is not nice to make the testcase OS specific.

          Unfortunately the proposed fix does not resolve the issue, the same error appears:

            TestTeraSort.testTeraSort:92->runTeraSort:67 ╗ IO No FileSystem for scheme: C
          

          If we don't want to make distinction in the testcase based on the OS then we need to extend the URI/Scheme
          validator to expect windows drive letters in front of the path.

          Please let me know your preferred solution.

          Thanks!

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Thanks Junping Du for the proposed enhancement. I agree that it is not nice to make the testcase OS specific. Unfortunately the proposed fix does not resolve the issue, the same error appears: TestTeraSort.testTeraSort:92->runTeraSort:67 ╗ IO No FileSystem for scheme: C If we don't want to make distinction in the testcase based on the OS then we need to extend the URI/Scheme validator to expect windows drive letters in front of the path. Please let me know your preferred solution. Thanks!
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Junping Du or Chris Nauroth: Could you please take a look on my previous comment and advise? Thanks!

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Junping Du or Chris Nauroth : Could you please take a look on my previous comment and advise? Thanks!
          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Tibor Kiss, thank you for finding this bug and providing a patch. I agree that ideally the fix should involve going through the Path class, because that already encapsulates the OS detection and special case handling for the Windows drive spec. Can you try testing the following?

            private static final String TEST_DIR = new Path(
                new File(System.getProperty("test.build.data", "/tmp"), "terasort")
                    .getAbsoluteFile().toURI()).toString();
          

          I expect this will work. Converting the File to URI explicitly captures that the scheme is "file:". Then, passing that URI through Path uses the Windows special-case path handling logic that we use throughout Hadoop.

          Show
          cnauroth Chris Nauroth added a comment - Hello Tibor Kiss , thank you for finding this bug and providing a patch. I agree that ideally the fix should involve going through the Path class, because that already encapsulates the OS detection and special case handling for the Windows drive spec. Can you try testing the following? private static final String TEST_DIR = new Path( new File( System .getProperty( "test.build.data" , "/tmp" ), "terasort" ) .getAbsoluteFile().toURI()).toString(); I expect this will work. Converting the File to URI explicitly captures that the scheme is "file:". Then, passing that URI through Path uses the Windows special-case path handling logic that we use throughout Hadoop.
          Hide
          tibor.kiss@gmail.com Tibor Kiss added a comment -

          Thanks Chris Nauroth, your proposed solution works like a charm on Windows & CentOS.

          I have attached a new patch to the ticket.

          Show
          tibor.kiss@gmail.com Tibor Kiss added a comment - Thanks Chris Nauroth , your proposed solution works like a charm on Windows & CentOS. I have attached a new patch to the ticket.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 14s trunk passed with JDK v1.8.0_92
          +1 compile 0m 15s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_92
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_92
          +1 javac 0m 11s the patch passed
          +1 compile 0m 14s the patch passed with JDK v1.7.0_95
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 1s Patch has no whitespace issues.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_92
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 24s hadoop-mapreduce-examples in the patch passed with JDK v1.8.0_92.
          +1 unit 0m 29s hadoop-mapreduce-examples in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          13m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801427/MAPREDUCE-6672.02.patch
          JIRA Issue MAPREDUCE-6672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux adec2ccce0bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4a85085
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6469/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project/hadoop-mapreduce-examples
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6469/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 14s trunk passed with JDK v1.8.0_92 +1 compile 0m 15s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_92 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 16s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_92 +1 javac 0m 11s the patch passed +1 compile 0m 14s the patch passed with JDK v1.7.0_95 +1 javac 0m 14s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 1s Patch has no whitespace issues. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_92 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 24s hadoop-mapreduce-examples in the patch passed with JDK v1.8.0_92. +1 unit 0m 29s hadoop-mapreduce-examples in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 13m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801427/MAPREDUCE-6672.02.patch JIRA Issue MAPREDUCE-6672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux adec2ccce0bd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4a85085 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6469/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project/hadoop-mapreduce-examples Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6469/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Tibor updating the patch and Chris proposing the solutions. 02 patch LGTM. +1.
          Will commit it later time today if no further comments from Chris on the patch.

          Show
          djp Junping Du added a comment - Thanks Tibor updating the patch and Chris proposing the solutions. 02 patch LGTM. +1. Will commit it later time today if no further comments from Chris on the patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I went ahead and committed to trunk, branch-2 and branch-2.8. Tibor Kiss, thank you for contributing the patch. Junping Du, thank you for code reviewing.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I went ahead and committed to trunk, branch-2 and branch-2.8. Tibor Kiss , thank you for contributing the patch. Junping Du , thank you for code reviewing.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9696 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9696/)
          MAPREDUCE-6672. TestTeraSort fails on Windows. Contributed by Tibor (cnauroth: rev 26ac0f8f0b28758a6404cd7b4b81d79b61d809d9)

          • hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9696 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9696/ ) MAPREDUCE-6672 . TestTeraSort fails on Windows. Contributed by Tibor (cnauroth: rev 26ac0f8f0b28758a6404cd7b4b81d79b61d809d9) hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/examples/terasort/TestTeraSort.java

            People

            • Assignee:
              tibor.kiss@gmail.com Tibor Kiss
              Reporter:
              tibor.kiss@gmail.com Tibor Kiss
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development