Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      Executing this code:

      Path file = new Path("file:///", "C:/trunk/build/test/data"); or Path file = new Path("C:/trunk/build/test/data");
      FileSystem fs = file.getFileSystem(conf);
      fs.mkdirs(file))

      produces this exception. It looks like it's defaulting to the DistributedFileSystem.

      2007-02-14 11:36:31,700 INFO mapred.TaskInProgress (TaskInProgress.java:updateStatus(296)) - Error from task_0001_m_000000_0: java.lang.IllegalArgumentException: Pathname /C:/trunk/build/test/data from C:/trunk/build/test/data is not a valid DFS filename.
      at org.apache.hadoop.dfs.DistributedFileSystem.getPath(DistributedFileSystem.java:111)
      at org.apache.hadoop.dfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:211)
      at org.apache.hadoop.mapred.MRCaching$MapClass.configure(MRCaching.java:68)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:50)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:70)
      at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:50)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:70)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:178)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1396)

      1. TestPath.java.patch
        1 kB
        Nigel Daley
      2. HADOOP-1020-2.patch
        3 kB
        Doug Cutting
      3. HADOOP-1020.patch
        2 kB
        Doug Cutting

        Activity

        Hide
        Doug Cutting added a comment -

        Is this on Windows? The Path code behaves differently on Windows.

        Can you provide a standalone test? Adding something to TestPath might be appropriate.

        Show
        Doug Cutting added a comment - Is this on Windows? The Path code behaves differently on Windows. Can you provide a standalone test? Adding something to TestPath might be appropriate.
        Hide
        Doug Cutting added a comment -

        A unit test might simply check that, e.g.: fs == FileSystem.getLocal(conf).

        Show
        Doug Cutting added a comment - A unit test might simply check that, e.g.: fs == FileSystem.getLocal(conf).
        Hide
        Nigel Daley added a comment -

        The TestPath.java.patch attachment adds the suggested test case to TestPath. The last 2 assertions in TestPath.testGetFileSystem fail on both Windows and Linux:
        assertEquals(new Path("file:///","C:/foo/bar").getFileSystem(conf), FileSystem.getLocal(conf));
        assertEquals(new Path("file:///","/foo/bar").getFileSystem(conf), FileSystem.getLocal(conf));
        where the conf object has an "fs.default.name" set to a dfs filesytem.

        The second assertion on Linux and both on Windows fails with:
        expected:<DFS[org.apache.hadoop.dfs.DFSClient@1cd66ea]> but was:<LocalFS>
        junit.framework.AssertionFailedError: expected:<DFS[org.apache.hadoop.dfs.DFSClient@1cd66ea]> but was:<LocalFS>
        at org.apache.hadoop.fs.TestPath.testGetFileSystem(TestPath.java:145)

        The first assertion on Linux fails with:
        No FileSystem for scheme: C
        java.io.IOException: No FileSystem for scheme: C
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:163)
        at org.apache.hadoop.fs.TestPath.testGetFileSystem(TestPath.java:145)

        Given the lack of specification in Path, the expected behaviour is not clear.

        Show
        Nigel Daley added a comment - The TestPath.java.patch attachment adds the suggested test case to TestPath. The last 2 assertions in TestPath.testGetFileSystem fail on both Windows and Linux: assertEquals(new Path("file:///","C:/foo/bar").getFileSystem(conf), FileSystem.getLocal(conf)); assertEquals(new Path("file:///","/foo/bar").getFileSystem(conf), FileSystem.getLocal(conf)); where the conf object has an "fs.default.name" set to a dfs filesytem. The second assertion on Linux and both on Windows fails with: expected:<DFS [org.apache.hadoop.dfs.DFSClient@1cd66ea] > but was:<LocalFS> junit.framework.AssertionFailedError: expected:<DFS [org.apache.hadoop.dfs.DFSClient@1cd66ea] > but was:<LocalFS> at org.apache.hadoop.fs.TestPath.testGetFileSystem(TestPath.java:145) The first assertion on Linux fails with: No FileSystem for scheme: C java.io.IOException: No FileSystem for scheme: C at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:163) at org.apache.hadoop.fs.TestPath.testGetFileSystem(TestPath.java:145) Given the lack of specification in Path, the expected behaviour is not clear.
        Hide
        Doug Cutting added a comment -

        Windows pathnames are problematic as URIs. For back-compatibility, I special-cased things on Windows so that a path that begins with <letter><colon> is treated as though it begins file:///<letter><colon>. This handling is disabled on other operating systems. Perhaps we should disable it altogether, but then I fear lots of things would break on Windows.

        That explains the first assertion failing on Linux. The failure of the second assertion looks like a bug to me.

        Show
        Doug Cutting added a comment - Windows pathnames are problematic as URIs. For back-compatibility, I special-cased things on Windows so that a path that begins with <letter><colon> is treated as though it begins file:/// <letter><colon>. This handling is disabled on other operating systems. Perhaps we should disable it altogether, but then I fear lots of things would break on Windows. That explains the first assertion failing on Linux. The failure of the second assertion looks like a bug to me.
        Hide
        Doug Cutting added a comment -

        Jira mangled the second occurrence of <letter><colon> in the previous comment.

        Show
        Doug Cutting added a comment - Jira mangled the second occurrence of <letter><colon> in the previous comment.
        Hide
        Doug Cutting added a comment -

        The bug was an erroneous optimization in the Path(Path,Path) constructor. I removed that and added a test case in the attached patch.

        Nigel, does this fix things for you on Windows?

        Show
        Doug Cutting added a comment - The bug was an erroneous optimization in the Path(Path,Path) constructor. I removed that and added a test case in the attached patch. Nigel, does this fix things for you on Windows?
        Hide
        Hadoop QA added a comment -

        -1, because 2 attempts failed to build and test the latest attachment (http://issues.apache.org/jira/secure/attachment/12352066/HADOOP-1020.patch) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512006. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch

        Show
        Hadoop QA added a comment - -1, because 2 attempts failed to build and test the latest attachment ( http://issues.apache.org/jira/secure/attachment/12352066/HADOOP-1020.patch ) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512006 . Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
        Hide
        Hadoop QA added a comment -

        -1, because 2 attempts failed to build and test the latest attachment (http://issues.apache.org/jira/secure/attachment/12352066/HADOOP-1020.patch</a>) against trunk revision <a href=) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512006. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch

        Show
        Hadoop QA added a comment - -1, because 2 attempts failed to build and test the latest attachment ( http://issues.apache.org/jira/secure/attachment/12352066/HADOOP-1020.patch </a>) against trunk revision <a href=) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512006 . Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
        Hide
        Doug Cutting added a comment -

        The problem with TestSymLink was that hadoop.tmp.dir had c:/... in it, and this is by default used as the basis for hadoop.system.dir, but it's not a valid HDFS path, since it contains a colon, and the system dir is located in the default filesystem, HDFS in this case.

        Here's a new version of that patch that fixes that by explicitly specifying a relative path for hadoop.system.dir in contrib tests.

        Show
        Doug Cutting added a comment - The problem with TestSymLink was that hadoop.tmp.dir had c:/... in it, and this is by default used as the basis for hadoop.system.dir, but it's not a valid HDFS path, since it contains a colon, and the system dir is located in the default filesystem, HDFS in this case. Here's a new version of that patch that fixes that by explicitly specifying a relative path for hadoop.system.dir in contrib tests.
        Hide
        Hadoop QA added a comment -

        -1, because javac generated 768 warnings (more than the acceptable 766 warnings) when testing the latest attachment (http://issues.apache.org/jira/secure/attachment/12352189/HADOOP-1020-2.patch) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512461. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch

        Show
        Hadoop QA added a comment - -1, because javac generated 768 warnings (more than the acceptable 766 warnings) when testing the latest attachment ( http://issues.apache.org/jira/secure/attachment/12352189/HADOOP-1020-2.patch ) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512461 . Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
        Hide
        Hadoop QA added a comment -

        -1, because 3 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12352189/HADOOP-1020-2.patch</a>) against trunk revision <a href= against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512499. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch

        Show
        Hadoop QA added a comment - -1, because 3 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12352189/HADOOP-1020-2.patch </a>) against trunk revision <a href= against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512499 . Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
        Show
        Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12352189/HADOOP-1020-2.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512499 . Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
        Hide
        Doug Cutting added a comment -

        I just committed this.

        Show
        Doug Cutting added a comment - I just committed this.

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Nigel Daley
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development