Hadoop Common
  1. Hadoop Common
  2. HADOOP-6286

The Glob methods in FileContext doe not deal with URIs correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The glob methods in FileContext were copied from FileSystem where they dealt with the a single filesystem.
      While they were extended to fit in FileContext they don't seem to deal with URI pathnames correctly.

      For example the following two lines in FileContext may be the source of the problem - the scheme and authority seem to be
      ignored beyond these points.

      line 1013: String filename = pathPattern.toUri().getPath();
      line 1035: String filename = pathPattern.toUri().getPath();

      1. HADOOP-6286.patch
        1 kB
        Boris Shkolnik
      2. HADOOP-6286-1.patch
        2 kB
        Boris Shkolnik
      3. HADOOP-6286-1.patch
        2 kB
        Boris Shkolnik
      4. HADOOP-6286-2.patch
        5 kB
        Boris Shkolnik
      5. HADOOP-6286-3.patch
        4 kB
        Boris Shkolnik
      6. HADOOP-6286-4.patch
        4 kB
        Boris Shkolnik

        Activity

        Hide
        Boris Shkolnik added a comment -

        removed system.out

        Show
        Boris Shkolnik added a comment - removed system.out
        Hide
        Boris Shkolnik added a comment -

        took different approach - passing URI FS into globInternal to be able to restore full abs path.

        Show
        Boris Shkolnik added a comment - took different approach - passing URI FS into globInternal to be able to restore full abs path.
        Hide
        Boris Shkolnik added a comment -

        pass URI instead of FS to globInternal

        Show
        Boris Shkolnik added a comment - pass URI instead of FS to globInternal
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12421449/HADOOP-6286-3.patch
        against trunk revision 820094.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421449/HADOOP-6286-3.patch against trunk revision 820094. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/72/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment -

        cleaned up patch

        Show
        Boris Shkolnik added a comment - cleaned up patch
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12421469/HADOOP-6286-4.patch
        against trunk revision 820094.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421469/HADOOP-6286-4.patch against trunk revision 820094. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/73/console This message is automatically generated.
        Hide
        Sanjay Radia added a comment -

        +1.
        Add tests as we discussed.

        Show
        Sanjay Radia added a comment - +1. Add tests as we discussed.
        Hide
        Boris Shkolnik added a comment -

        current tests for FileSystem.glob are in HDFS
        I am going to add test for FileContext there. (HDFS-678)

        Show
        Boris Shkolnik added a comment - current tests for FileSystem.glob are in HDFS I am going to add test for FileContext there. ( HDFS-678 )
        Hide
        Suresh Srinivas added a comment -

        I committed this to both trunk and 21 branch. Thanks you Boris.

        Show
        Suresh Srinivas added a comment - I committed this to both trunk and 21 branch. Thanks you Boris.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #54 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/54/)
        . Fix bugs in related to URI handling in glob methods in FileContext. Contributed by Boris Shkolnik.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #54 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/54/ ) . Fix bugs in related to URI handling in glob methods in FileContext. Contributed by Boris Shkolnik.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #121 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/121/)
        . Fix bugs in related to URI handling in glob methods in FileContext. Contributed by Boris Shkolnik.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #121 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/121/ ) . Fix bugs in related to URI handling in glob methods in FileContext. Contributed by Boris Shkolnik.

          People

          • Assignee:
            Boris Shkolnik
            Reporter:
            Sanjay Radia
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development