Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8977

multiple FsShell test failures on Windows

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • trunk-win
    • trunk-win
    • fs
    • None
    • Reviewed

    Description

      Multiple FsShell-related tests fail on Windows. Commands are returning non-zero exit status.

      Attachments

        1. HADOOP-8977-branch-trunk-win.patch
          10 kB
          Chris Nauroth
        2. HADOOP-8977-branch-trunk-win.patch
          12 kB
          Chris Nauroth
        3. HADOOP-8977-branch-trunk-win.patch
          11 kB
          Chris Nauroth
        4. HADOOP-8977.patch
          11 kB
          Suresh Srinivas

        Issue Links

          Activity

            cnauroth Chris Nauroth added a comment -

            This includes the following test suites:

            org.apache.hadoop.fs.TestFsShellCopy
            org.apache.hadoop.fs.TestFsShellReturnCode

            Tests run: 8, Failures: 7, Errors: 0, Skipped: 0, Time elapsed: 0.813 sec <<< FAILURE!
            testCopyNoCrc(org.apache.hadoop.fs.TestFsShellCopy)  Time elapsed: 109 sec  <<< FAILURE!
            java.lang.AssertionError: expected:<0> but was:<-1>
            	at org.junit.Assert.fail(Assert.java:91)
            	at org.junit.Assert.failNotEquals(Assert.java:645)
            	at org.junit.Assert.assertEquals(Assert.java:126)
            	at org.junit.Assert.assertEquals(Assert.java:470)
            	at org.junit.Assert.assertEquals(Assert.java:454)
            	at org.apache.hadoop.fs.TestFsShellCopy.shellRun(TestFsShellCopy.java:97)
            	at org.apache.hadoop.fs.TestFsShellCopy.testCopyNoCrc(TestFsShellCopy.java:65)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
            	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
            	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
            	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
            	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
            	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
            	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
            	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
            	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
            	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
            	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
            	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
            
            Running org.apache.hadoop.fs.TestFsShellReturnCode
            Tests run: 6, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 1.734 sec <<< FAILURE!
            testChmod(org.apache.hadoop.fs.TestFsShellReturnCode)  Time elapsed: 94 sec  <<< FAILURE!
            java.lang.AssertionError: expected:<1> but was:<-1>
            	at org.junit.Assert.fail(Assert.java:91)
            	at org.junit.Assert.failNotEquals(Assert.java:645)
            	at org.junit.Assert.assertEquals(Assert.java:126)
            	at org.junit.Assert.assertEquals(Assert.java:470)
            	at org.junit.Assert.assertEquals(Assert.java:454)
            	at org.apache.hadoop.fs.TestFsShellReturnCode.testChmod(TestFsShellReturnCode.java:149)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
            	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
            	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
            	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
            	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
            	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
            	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
            	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
            	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
            	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
            	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
            	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
            	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
            	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
            	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
            	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
            	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
            
            cnauroth Chris Nauroth added a comment - This includes the following test suites: org.apache.hadoop.fs.TestFsShellCopy org.apache.hadoop.fs.TestFsShellReturnCode Tests run: 8, Failures: 7, Errors: 0, Skipped: 0, Time elapsed: 0.813 sec <<< FAILURE! testCopyNoCrc(org.apache.hadoop.fs.TestFsShellCopy) Time elapsed: 109 sec <<< FAILURE! java.lang.AssertionError: expected:<0> but was:<-1> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.junit.Assert.assertEquals(Assert.java:454) at org.apache.hadoop.fs.TestFsShellCopy.shellRun(TestFsShellCopy.java:97) at org.apache.hadoop.fs.TestFsShellCopy.testCopyNoCrc(TestFsShellCopy.java:65) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Running org.apache.hadoop.fs.TestFsShellReturnCode Tests run: 6, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 1.734 sec <<< FAILURE! testChmod(org.apache.hadoop.fs.TestFsShellReturnCode) Time elapsed: 94 sec <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<-1> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.junit.Assert.assertEquals(Assert.java:454) at org.apache.hadoop.fs.TestFsShellReturnCode.testChmod(TestFsShellReturnCode.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
            cnauroth Chris Nauroth added a comment -

            The attached patch fixes the tests. The tests pass on both Mac and Windows. There were 2 separate issues.

            For TestFsShellReturnCode, it was another case of test code creating invalid test paths by manually concatenating strings that mingle '/' and '\' as path separators. These are all fixed by switching to the Path constructor that takes parent and child explicitly as arguments.

            For TestFsShellCopy, there was a bug in CommandWithDestination and its subclasses. They were using a PathData constructor that accepts a File object. This constructor was meant to indicate that the file's path is interpreted as local. This doesn't work on Windows though, because File.getPath() and File.toString() will convert '/' to '\', and the downstream code doesn't expect this. To fix this, I removed the dangerous PathData constructor. Then, I switched from private to public for the constructor that accepts a FileSystem argument. Then, I changed all call sites to use this constructor with a local FileSystem to preserve the semantics that we are talking about local paths. PathData is @InterfaceAudience.Private and @InterfaceStability.Unstable, so I expect the interface change to be safe.

            This change is dependent on HADOOP-8953, which also changes PathData. My patch is relative to that patch.

            cnauroth Chris Nauroth added a comment - The attached patch fixes the tests. The tests pass on both Mac and Windows. There were 2 separate issues. For TestFsShellReturnCode, it was another case of test code creating invalid test paths by manually concatenating strings that mingle '/' and '\' as path separators. These are all fixed by switching to the Path constructor that takes parent and child explicitly as arguments. For TestFsShellCopy, there was a bug in CommandWithDestination and its subclasses. They were using a PathData constructor that accepts a File object. This constructor was meant to indicate that the file's path is interpreted as local. This doesn't work on Windows though, because File.getPath() and File.toString() will convert '/' to '\', and the downstream code doesn't expect this. To fix this, I removed the dangerous PathData constructor. Then, I switched from private to public for the constructor that accepts a FileSystem argument. Then, I changed all call sites to use this constructor with a local FileSystem to preserve the semantics that we are talking about local paths. PathData is @InterfaceAudience.Private and @InterfaceStability.Unstable, so I expect the interface change to be safe. This change is dependent on HADOOP-8953 , which also changes PathData. My patch is relative to that patch.
            daryn Daryn Sharp added a comment -

            The ctors that takes a FileSystem are intentional not exposed, so please don't make them public. I think all that may be required is to change file.toString() to file.toUri().toString().

            daryn Daryn Sharp added a comment - The ctors that takes a FileSystem are intentional not exposed, so please don't make them public. I think all that may be required is to change file.toString() to file.toUri().toString() .
            cnauroth Chris Nauroth added a comment -

            Thanks, Daryn. I tested multiple different attempts at translating the File back to a workable path, including File.toUri().toString(). Unfortunately, on Windows, taking a trip through java.io.File always converts '/' to '\', no matter what additional API calls you make on it.

            I'm attaching an alternative patch that keeps the PathData(FileSystem) constructor private like you requested, but changes the public PathData(File) constructor to PathData(URI) and updates all call sites. URI doesn't suffer from the same problem as File. This passes on Mac and Windows.

            What do you think of this version of the patch?

            cnauroth Chris Nauroth added a comment - Thanks, Daryn. I tested multiple different attempts at translating the File back to a workable path, including File.toUri().toString(). Unfortunately, on Windows, taking a trip through java.io.File always converts '/' to '\', no matter what additional API calls you make on it. I'm attaching an alternative patch that keeps the PathData(FileSystem) constructor private like you requested, but changes the public PathData(File) constructor to PathData(URI) and updates all call sites. URI doesn't suffer from the same problem as File. This passes on Mac and Windows. What do you think of this version of the patch?
            cnauroth Chris Nauroth added a comment -

            Hi, Daryn. Do you have any additional feedback on the new version of the patch that I uploaded on 10/31? Thank you.

            cnauroth Chris Nauroth added a comment - Hi, Daryn. Do you have any additional feedback on the new version of the patch that I uploaded on 10/31? Thank you.
            arp Arpit Agarwal added a comment -

            I am taking a look to see if I can help get any traction.

            arp Arpit Agarwal added a comment - I am taking a look to see if I can help get any traction.
            cnauroth Chris Nauroth added a comment -

            Updating patch to remove a System.out.println call that I added by mistake.

            cnauroth Chris Nauroth added a comment - Updating patch to remove a System.out.println call that I added by mistake.
            arp Arpit Agarwal added a comment -

            The PathData public constructor change in the 10/31 patch looks okay to me since the interface is labeled private and unstable. Unfortunate that File mangles the forward slashes.

            +1 (non-binding). Reviewed and ran the following tests which all passed:

            • TestFsShellCopy
            • TestFsShellReturnCode
            • TestTextCommand
            • TestPath
            • TestPathData
            arp Arpit Agarwal added a comment - The PathData public constructor change in the 10/31 patch looks okay to me since the interface is labeled private and unstable. Unfortunate that File mangles the forward slashes. +1 (non-binding). Reviewed and ran the following tests which all passed: TestFsShellCopy TestFsShellReturnCode TestTextCommand TestPath TestPathData

            patch looks good. Minor nits - use "} catch {" per coding convention

            sureshms Suresh Srinivas added a comment - patch looks good. Minor nits - use "} catch {" per coding convention

            Updated patch with minor nit taken care of.

            sureshms Suresh Srinivas added a comment - Updated patch with minor nit taken care of.

            I committed the patch. Thank you Chris. Thank you Arpit for the review.

            sureshms Suresh Srinivas added a comment - I committed the patch. Thank you Chris. Thank you Arpit for the review.

            BTW I missed the following comment made earlier:

            Hi, Daryn. Do you have any additional feedback on the new version of the patch that I uploaded on 10/31? Thank you.

            Daryn, please do post any further comments you may have. That could be addressed in later jiras.

            sureshms Suresh Srinivas added a comment - BTW I missed the following comment made earlier: Hi, Daryn. Do you have any additional feedback on the new version of the patch that I uploaded on 10/31? Thank you. Daryn, please do post any further comments you may have. That could be addressed in later jiras.
            daryn Daryn Sharp added a comment -

            Here's a copy-n-paste from an offline discussion:

            The problem here is hadoop is designed around URIs and we are trying to support something (windows paths) that is NOT a URI. Earlier, I bent over backwards trying to temporarily handle windows paths but Suresh convinced me it was the wrong direction.

            The article I linked is about how to properly reference windows paths as URIs and says windows style \ paths are deprecated in IE which I think essentially means the file browser. The windows shell supports / paths so I'm grappling with why we should perpetuate deprecated windows paths as pseudo-URIs when real URIs appear to be fully supported in windows.

            I'd be a bit happier ( or less unhappy! )if \ support is more context specific to just windows local path names. As it stands, all URIs on windows are subject to \ to / conversion which prevents windows from accessing valid filenames in hdfs and other supported filesystems. I can understand/sympathize with the motivation to support c:\path, but I don't agree that hdfs:\\host\path, or hdfs:\/path/path2\path3 should be supported at all. This bizarre behavior creates compatibility issues where jobs accessing paths in that way are not cross-platform compatible. Ie. They "work" on hadoop for windows, but fail on every other OS. Once we "let the cat of of the bag" by adding more pseudo-support for non-URIs on windows, it's going to be that much harder to take it away.

            What if we did something a bit more selective:

            1. [a-z]:\ considered a windows non-URI
              • implicitly deemed to have a "file" scheme if not already declared
              • all \ are converted to / - which means no quoting of metachars available, or we support ^ as the escape
              • throw an exception if / already exists in the path
            2. [a-z]:/
              • considered a standard URI
              • implicitly deemed to have a "file" scheme if not already declared
              • no \ conversion - quoting of metachars is supported
            3. all other URI schemes and relative paths
              • no change
            4. add ctor Path(File)
              • allow users to create Paths from non-URIs
              • will eventually be the only supported way to access non-URI paths
            5. eliminate treating ":" as an invalid path character to allow drive letters

            I'm curious what serious breakage we'll have if we just require standard URIs - ie. change little to nothing or implement the above proposal?

            daryn Daryn Sharp added a comment - Here's a copy-n-paste from an offline discussion: The problem here is hadoop is designed around URIs and we are trying to support something (windows paths) that is NOT a URI. Earlier, I bent over backwards trying to temporarily handle windows paths but Suresh convinced me it was the wrong direction. The article I linked is about how to properly reference windows paths as URIs and says windows style \ paths are deprecated in IE which I think essentially means the file browser. The windows shell supports / paths so I'm grappling with why we should perpetuate deprecated windows paths as pseudo-URIs when real URIs appear to be fully supported in windows. I'd be a bit happier ( or less unhappy! )if \ support is more context specific to just windows local path names. As it stands, all URIs on windows are subject to \ to / conversion which prevents windows from accessing valid filenames in hdfs and other supported filesystems. I can understand/sympathize with the motivation to support c:\path, but I don't agree that hdfs:\\host\path, or hdfs:\/path/path2\path3 should be supported at all. This bizarre behavior creates compatibility issues where jobs accessing paths in that way are not cross-platform compatible. Ie. They "work" on hadoop for windows, but fail on every other OS. Once we "let the cat of of the bag" by adding more pseudo-support for non-URIs on windows, it's going to be that much harder to take it away. What if we did something a bit more selective: [a-z] :\ considered a windows non-URI implicitly deemed to have a "file" scheme if not already declared all \ are converted to / - which means no quoting of metachars available, or we support ^ as the escape throw an exception if / already exists in the path [a-z] :/ considered a standard URI implicitly deemed to have a "file" scheme if not already declared no \ conversion - quoting of metachars is supported all other URI schemes and relative paths no change add ctor Path(File) allow users to create Paths from non-URIs will eventually be the only supported way to access non-URI paths eliminate treating ":" as an invalid path character to allow drive letters I'm curious what serious breakage we'll have if we just require standard URIs - ie. change little to nothing or implement the above proposal?
            cnauroth Chris Nauroth added a comment -

            Linking to HADOOP-9018 for tracking of long-term goals.

            cnauroth Chris Nauroth added a comment - Linking to HADOOP-9018 for tracking of long-term goals.
            arp Arpit Agarwal added a comment -

            Daryn, I will address most of the issues you mention when I get to HADOOP-9018 next week.

            arp Arpit Agarwal added a comment - Daryn, I will address most of the issues you mention when I get to HADOOP-9018 next week.
            vinayakumarb Vinayakumar B added a comment -

            HDFS-13911 seems to be broken by this change.

            After this change, all local paths with special characters (ex: space) should be encoded first before passing as argument to command line.

            vinayakumarb Vinayakumar B added a comment - HDFS-13911 seems to be broken by this change. After this change, all local paths with special characters (ex: space) should be encoded first before passing as argument to command line.

            People

              cnauroth Chris Nauroth
              cnauroth Chris Nauroth
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: