Hadoop Common
  1. Hadoop Common
  2. HADOOP-7418

support for multiple slashes in the path separator

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      Linux running JDK 1.6

    • Tags:
      Path issue

      Description

      the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
      //user/directory1//directory2.

      While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.

      Exception in thread "main" java.net.UnknownHostException: unknown host: user
      at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)

      as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

      1. HDFS-1460.txt
        3 kB
        Andrew Look
      2. HDFS-1460.txt
        3 kB
        Andrew Look
      3. HADOOP-7418.txt
        3 kB
        Andrew Look
      4. HADOOP-7418.txt
        4 kB
        Andrew Look
      5. HADOOP-7418--20110719.txt
        4 kB
        Andrew Look

        Issue Links

          Activity

          Hide
          Andrew Look added a comment -

          Attached is a patch for the issue in question.

          Note that I made a change in the way toString() is rendered that could potentially break backwards compatibility. As a result, I modified three assertions in TestPath.testNormalize() in order to correctly interpret the absolute pathnames as canonical paths.

          For instance, consider the assertion:
          assertEquals("//foo/", new Path("//foo/").toString());

          The two slashes at the beginning of the string are only rendered because the URI is created with "foo" as the authority; if those two slashes were part of the path, they would get normalized to a single slash. Also, the trailing slash would have been trimmed if the whole string had been considered a path; the trailing slash is only inserted after an authority.

          Therefore, if we are not treating "foo" as the hostname/authority but instead as the first directory in a path, then new Path("//foo/").toString() should be equal to "/foo".

          Hopefully no one was depending on the first directory being an authority in a scheme-less URI string, otherwise this might break compatibility. If URI's are being used correctly, however, this should be okay.

          Show
          Andrew Look added a comment - Attached is a patch for the issue in question. Note that I made a change in the way toString() is rendered that could potentially break backwards compatibility. As a result, I modified three assertions in TestPath.testNormalize() in order to correctly interpret the absolute pathnames as canonical paths. For instance, consider the assertion: assertEquals("//foo/", new Path("//foo/").toString()); The two slashes at the beginning of the string are only rendered because the URI is created with "foo" as the authority; if those two slashes were part of the path, they would get normalized to a single slash. Also, the trailing slash would have been trimmed if the whole string had been considered a path; the trailing slash is only inserted after an authority. Therefore, if we are not treating "foo" as the hostname/authority but instead as the first directory in a path, then new Path("//foo/").toString() should be equal to "/foo". Hopefully no one was depending on the first directory being an authority in a scheme-less URI string, otherwise this might break compatibility. If URI's are being used correctly, however, this should be okay.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483524/HDFS-1460.txt
          against trunk revision 1138645.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/824//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483524/HDFS-1460.txt against trunk revision 1138645. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/824//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483526/HDFS-1460.txt
          against trunk revision 1138645.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/825//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483526/HDFS-1460.txt against trunk revision 1138645. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/825//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483526/HDFS-1460.txt
          against trunk revision 1137724.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483526/HDFS-1460.txt against trunk revision 1137724. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Hey Andrew, patch looks pretty good to me. One question:

          From manually inspecting the code, I suspect the Path class will still not correctly handle the case of multiple trailing slashes. Would you mind adding a test case to test some path with a silly number (> 5) of trailing slashes?

          You might also consider running the full HDFS and MR test suites with this patch installed, as those obviously exercise all of the Path code quite a bit. See the "Changes that span projects" section of this page for instructions about how to get HDFS and MR to compile against your patched Common: http://wiki.apache.org/hadoop/HowToContribute

          Show
          Aaron T. Myers added a comment - Hey Andrew, patch looks pretty good to me. One question: From manually inspecting the code, I suspect the Path class will still not correctly handle the case of multiple trailing slashes. Would you mind adding a test case to test some path with a silly number (> 5) of trailing slashes? You might also consider running the full HDFS and MR test suites with this patch installed, as those obviously exercise all of the Path code quite a bit. See the "Changes that span projects" section of this page for instructions about how to get HDFS and MR to compile against your patched Common: http://wiki.apache.org/hadoop/HowToContribute
          Hide
          Andrew Look added a comment -

          I redid my patch to handle the case of >2 slashes, and added a couple of unit tests as you suggested.

          When running the test suite, I had some tests hang due to timeouts, but no failures. I tried running the pre-patch versions of HDFS's test suite, and for instance, TestHDFSCLI times out because it can't connect to a namenode. Similar outcomes for a number of other tests: TestLargeBlock, TestReplaceDataNodeOnFailure...

          Will the Jenkins instance run these for me?

          Show
          Andrew Look added a comment - I redid my patch to handle the case of >2 slashes, and added a couple of unit tests as you suggested. When running the test suite, I had some tests hang due to timeouts, but no failures. I tried running the pre-patch versions of HDFS's test suite, and for instance, TestHDFSCLI times out because it can't connect to a namenode. Similar outcomes for a number of other tests: TestLargeBlock, TestReplaceDataNodeOnFailure... Will the Jenkins instance run these for me?
          Hide
          Aaron T. Myers added a comment -

          Will the Jenkins instance run these for me?

          Jenkins will only run the tests for the Common sub-project, since this is a JIRA for Common.

          I'll take a look at this patch and try to run some of tests you mention that timed out. We see a few notoriously flaky tests fail spuriously pretty often, but not usually these, and not usually with timeouts.

          Show
          Aaron T. Myers added a comment - Will the Jenkins instance run these for me? Jenkins will only run the tests for the Common sub-project, since this is a JIRA for Common. I'll take a look at this patch and try to run some of tests you mention that timed out. We see a few notoriously flaky tests fail spuriously pretty often, but not usually these, and not usually with timeouts.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12484565/HADOOP-7418.txt
          against trunk revision 1140442.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12484565/HADOOP-7418.txt against trunk revision 1140442. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console This message is automatically generated.
          Hide
          Andrew Look added a comment -

          Great, thanks Aaron!

          I'm running OS X Snow Leopard so that may be causing problems on my end... I've heard of others having trouble building the C++ libraries in HDFS (see http://cxwangyi.blogspot.com/2010_04_01_archive.html ) and apparently some mods need to be made to get things to run correctly.

          Have you heard of any such issues on OS X?

          Show
          Andrew Look added a comment - Great, thanks Aaron! I'm running OS X Snow Leopard so that may be causing problems on my end... I've heard of others having trouble building the C++ libraries in HDFS (see http://cxwangyi.blogspot.com/2010_04_01_archive.html ) and apparently some mods need to be made to get things to run correctly. Have you heard of any such issues on OS X?
          Hide
          Aaron T. Myers added a comment -

          Have you heard of any such issues on OS X?

          There are a few known issues with native code compilation on OS X, but it's unlikely this would be the issue since you probably weren't compiling the native code when running the tests.

          I've personally run the tests you identified as failing on my machine with this patch and they pass just fine.

          Two small comments on the latest patch:

          1. Rather than a while loop to replace "//", how about just a regex? I think 'path.replaceAll("/+", "/")' should do the trick.
          2. Seems like we should also be adding similar code to deal with consecutive back slashes (as in Windows paths.)

          +1 once these are addressed. I'll commit it then if there are no other comments.

          Show
          Aaron T. Myers added a comment - Have you heard of any such issues on OS X? There are a few known issues with native code compilation on OS X, but it's unlikely this would be the issue since you probably weren't compiling the native code when running the tests. I've personally run the tests you identified as failing on my machine with this patch and they pass just fine. Two small comments on the latest patch: Rather than a while loop to replace "//", how about just a regex? I think ' path.replaceAll("/+", "/") ' should do the trick. Seems like we should also be adding similar code to deal with consecutive back slashes (as in Windows paths.) +1 once these are addressed. I'll commit it then if there are no other comments.
          Hide
          Andrew Look added a comment -

          I think I have the solution here... It seems that if there's only one backslash it doesn't necessarily get replaced.. do you think my regex for the backslashes is correct?

          Thanks for all your help!

          Show
          Andrew Look added a comment - I think I have the solution here... It seems that if there's only one backslash it doesn't necessarily get replaced.. do you think my regex for the backslashes is correct? Thanks for all your help!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485685/HADOOP-7418.txt
          against trunk revision 1144043.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.fs.TestPath

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485685/HADOOP-7418.txt against trunk revision 1144043. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.TestPath +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/711//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Hey Andrew, I think the regex needs to changed. In particular, I don't think it will actually cover the multiple back slash case since the double back slash in your regex actually is just string-escaping one back slash, which is then regex-escaping the "+" character. If you want to include a literal back slash in the regex, you need to use 4 back slashes. (Silly, I know.)

          Furthermore, I think that doing the replace in two stages (first forward slashes, then back slashes) won't cover the case when forward slashes are separated by back slashes (e.g. "/foo/\/bar".) To cover that case, you have two options:

          1. Replace back slashes first, before forward slashes. The back slash replacement could even be a 1-for-1 replacement, leaving you with a bunch of consecutive forward slashes, which then get replaced by a single forward slash in the next regex.
          2. Use something like this regex: ".replaceAll("(/|\\\\)+", "/")", which replaces multiple consecutive "/" or "\" with a single "/".

          It would also be worthwhile to add test cases to cover these cases.

          Show
          Aaron T. Myers added a comment - Hey Andrew, I think the regex needs to changed. In particular, I don't think it will actually cover the multiple back slash case since the double back slash in your regex actually is just string-escaping one back slash, which is then regex-escaping the "+" character. If you want to include a literal back slash in the regex, you need to use 4 back slashes. (Silly, I know.) Furthermore, I think that doing the replace in two stages (first forward slashes, then back slashes) won't cover the case when forward slashes are separated by back slashes (e.g. "/foo/\/bar".) To cover that case, you have two options: Replace back slashes first, before forward slashes. The back slash replacement could even be a 1-for-1 replacement, leaving you with a bunch of consecutive forward slashes, which then get replaced by a single forward slash in the next regex. Use something like this regex: " .replaceAll("(/|\\\\)+", "/") ", which replaces multiple consecutive "/" or "\" with a single "/". It would also be worthwhile to add test cases to cover these cases.
          Hide
          Andrew Look added a comment -

          Hi Aaron,

          I chose the first approach you recommended - replacing backslashes with slashes, then replacing multiple slashes with a single slash.

          I think that I added a good number of unit tests to verify this functionality; let me know if you think I need any more.

          Cheers (and thanks for all the help!)

          • Andrew
          Show
          Andrew Look added a comment - Hi Aaron, I chose the first approach you recommended - replacing backslashes with slashes, then replacing multiple slashes with a single slash. I think that I added a good number of unit tests to verify this functionality; let me know if you think I need any more. Cheers (and thanks for all the help!) Andrew
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487065/HADOOP-7418--20110719.txt
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487065/HADOOP-7418--20110719.txt against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/749//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I'm currently running the full HDFS test suite with this common patch in place to see if it catches any regressions. I'll commit this pending those results.

          Thanks a lot for addressing all my comments, Andrew.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I'm currently running the full HDFS test suite with this common patch in place to see if it catches any regressions. I'll commit this pending those results. Thanks a lot for addressing all my comments, Andrew.
          Hide
          Aaron T. Myers added a comment -

          Hey Andrew, turns out one HDFS test failed: TestFcHdfsSymlink. Would you mind looking into this?

          Show
          Aaron T. Myers added a comment - Hey Andrew, turns out one HDFS test failed: TestFcHdfsSymlink . Would you mind looking into this?
          Hide
          Harsh J added a comment -

          Andrew, did you get a chance to take a look at why TestFcHdfsSymlink may have failed? Would you still be interested in pursuing this patch?

          Show
          Harsh J added a comment - Andrew, did you get a chance to take a look at why TestFcHdfsSymlink may have failed? Would you still be interested in pursuing this patch?
          Hide
          Harsh J added a comment -

          Seems to still exist, though I'm unsure what exactly its trying to do:

          ➜  branch-2  hadoop fs -ls //user//harshchouraria/  
          12/09/23 01:53:32 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
          12/09/23 01:53:32 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=DEFAULT, sampleName=Ops)
          12/09/23 01:53:33 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
          12/09/23 01:53:33 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
          12/09/23 01:53:33 DEBUG security.Groups:  Creating new Groups object
          12/09/23 01:53:33 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
          2012-09-23 01:53:33.084 java[19004:1903] Unable to load realm info from SCDynamicStore
          12/09/23 01:53:33 DEBUG security.UserGroupInformation: hadoop login
          12/09/23 01:53:33 DEBUG security.UserGroupInformation: hadoop login commit
          12/09/23 01:53:33 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: harshchouraria
          12/09/23 01:53:33 DEBUG security.UserGroupInformation: UGI loginUser:harshchouraria (auth:SIMPLE)
          12/09/23 01:53:33 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWritable, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@3ff2cea2
          12/09/23 01:53:33 DEBUG hdfs.DFSClient: Short circuit read is false
          12/09/23 01:53:33 DEBUG ipc.Client: The ping interval is 60000 ms.
          12/09/23 01:53:33 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientNamenodeProtocolPB
          12/09/23 01:53:33 DEBUG ipc.Client: Connecting to localhost/127.0.0.1:8020
          12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria sending #0
          12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: starting, having connections 1
          12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria got value #0
          12/09/23 01:53:33 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo 69
          ls: `//user//harshchouraria/': No such file or directory
          12/09/23 01:53:33 DEBUG ipc.Client: Stopping client
          12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: closed
          12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: stopped, remaining connections 0
          

          The error isn't as bad as the one reported earlier though, I think.

          Show
          Harsh J added a comment - Seems to still exist, though I'm unsure what exactly its trying to do: ➜ branch-2 hadoop fs -ls //user//harshchouraria/ 12/09/23 01:53:32 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always= false , type=DEFAULT, sampleName=Ops) 12/09/23 01:53:32 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always= false , type=DEFAULT, sampleName=Ops) 12/09/23 01:53:33 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics 12/09/23 01:53:33 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty 12/09/23 01:53:33 DEBUG security.Groups: Creating new Groups object 12/09/23 01:53:33 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 2012-09-23 01:53:33.084 java[19004:1903] Unable to load realm info from SCDynamicStore 12/09/23 01:53:33 DEBUG security.UserGroupInformation: hadoop login 12/09/23 01:53:33 DEBUG security.UserGroupInformation: hadoop login commit 12/09/23 01:53:33 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: harshchouraria 12/09/23 01:53:33 DEBUG security.UserGroupInformation: UGI loginUser:harshchouraria (auth:SIMPLE) 12/09/23 01:53:33 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWritable, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@3ff2cea2 12/09/23 01:53:33 DEBUG hdfs.DFSClient: Short circuit read is false 12/09/23 01:53:33 DEBUG ipc.Client: The ping interval is 60000 ms. 12/09/23 01:53:33 DEBUG ipc.Client: Use SIMPLE authentication for protocol ClientNamenodeProtocolPB 12/09/23 01:53:33 DEBUG ipc.Client: Connecting to localhost/127.0.0.1:8020 12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria sending #0 12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: starting, having connections 1 12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria got value #0 12/09/23 01:53:33 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo 69 ls: ` //user//harshchouraria/': No such file or directory 12/09/23 01:53:33 DEBUG ipc.Client: Stopping client 12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: closed 12/09/23 01:53:33 DEBUG ipc.Client: IPC Client (371838604) connection to localhost/127.0.0.1:8020 from harshchouraria: stopped, remaining connections 0 The error isn't as bad as the one reported earlier though, I think.
          Hide
          Kiran Kumar M R added a comment -

          Current behavior of trunk and 7.2 is if // appears in the beginning of path, it is expected to have scheme: before that. So restriction is that // is not supported only in beginning of path as its used to identify scheme.

          Samples:

          >hadoop fs -ls //wc-out//_temporary
          ls: No FileSystem for scheme: null
          

          // is allowed later in path. Below path is valid

          >hadoop fs -ls /wc-out//_temporary
          Found 1 items
          drwxr-xr-x   - KiranKumar supergroup          0 2015-05-04 02:41 /wc-out/_temporary/1
          

          // is also valid if path has proper scheme

          >hadoop fs -ls hdfs://kiran-hpenvy//wc-out//_temporary
          Found 1 items
          drwxr-xr-x   - KiranKumar supergroup          0 2015-05-04 02:41 hdfs://kiran-hpenvy/wc-out/_temporary/1
          

          I think existing path behavior is fine. This issue can be closed.
          Others please comment.

          Show
          Kiran Kumar M R added a comment - Current behavior of trunk and 7.2 is if // appears in the beginning of path, it is expected to have scheme: before that. So restriction is that // is not supported only in beginning of path as its used to identify scheme. Samples: >hadoop fs -ls //wc-out//_temporary ls: No FileSystem for scheme: null // is allowed later in path. Below path is valid >hadoop fs -ls /wc-out //_temporary Found 1 items drwxr-xr-x - KiranKumar supergroup 0 2015-05-04 02:41 /wc-out/_temporary/1 // is also valid if path has proper scheme >hadoop fs -ls hdfs: //kiran-hpenvy//wc-out//_temporary Found 1 items drwxr-xr-x - KiranKumar supergroup 0 2015-05-04 02:41 hdfs: //kiran-hpenvy/wc-out/_temporary/1 I think existing path behavior is fine. This issue can be closed. Others please comment.
          Hide
          Uma Maheswara Rao G added a comment -

          Kiran Kumar M R, I think your analysis make sense to me. we can close this JIRA now. If others has different opinion feel free to open and then we can discuss further.

          Show
          Uma Maheswara Rao G added a comment - Kiran Kumar M R , I think your analysis make sense to me. we can close this JIRA now. If others has different opinion feel free to open and then we can discuss further.
          Hide
          Kiran Kumar M R added a comment -

          Just found this comment from Doug Cutting in HADOOP-8087 https://issues.apache.org/jira/browse/HADOOP-8087?focusedCommentId=13210604&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13210604

          URI RFC are quoted in that comment. Looks like we need to treat //dir1//dir2 as default scheme.
          Re opening.

          Show
          Kiran Kumar M R added a comment - Just found this comment from Doug Cutting in HADOOP-8087 https://issues.apache.org/jira/browse/HADOOP-8087?focusedCommentId=13210604&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13210604 URI RFC are quoted in that comment. Looks like we need to treat //dir1//dir2 as default scheme. Re opening.
          Hide
          Uma Maheswara Rao G added a comment -

          I think we need to work on as I saw the more discussions in comment, Let's reopen and duplicate one.

          Show
          Uma Maheswara Rao G added a comment - I think we need to work on as I saw the more discussions in comment , Let's reopen and duplicate one.
          Hide
          Kiran Kumar M R added a comment -

          HADOOP-8087 had patch attached recently. Fix for this issue can discussed and closed in that.

          Show
          Kiran Kumar M R added a comment - HADOOP-8087 had patch attached recently. Fix for this issue can discussed and closed in that.

            People

            • Assignee:
              Unassigned
              Reporter:
              Sudharsan Sampath
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development