Hadoop Common
  1. Hadoop Common
  2. HADOOP-6334

GenericOptionsParser does not understand uri for -files -libjars and -archives option

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If we give an uri for -files, -libjars and -archives option , in GenericOptionsParser. It gives FileNotFoundException.

      1. patch-6334-4.txt
        14 kB
        Amareshwari Sriramadasu
      2. patch-6334-3.txt
        14 kB
        Amareshwari Sriramadasu
      3. patch-6334-2.txt
        9 kB
        Amareshwari Sriramadasu
      4. patch-6334-1.txt
        14 kB
        Amareshwari Sriramadasu
      5. patch-6334.txt
        14 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Amareshwari Sriramadasu added a comment -

          Patch fixes org.apache.hadoop.fs.Path to understand fragment of the uri.
          Fixed GenericOptionsParser to throw out IOException, if any.
          Added corresponding testcases.

          Show
          Amareshwari Sriramadasu added a comment - Patch fixes org.apache.hadoop.fs.Path to understand fragment of the uri. Fixed GenericOptionsParser to throw out IOException, if any. Added corresponding testcases.
          Hide
          Amareshwari Sriramadasu added a comment -

          Fixed a minor bug in earlier patch. Updated the testcase.

          Show
          Amareshwari Sriramadasu added a comment - Fixed a minor bug in earlier patch. Updated the testcase.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423304/patch-6334.txt
          against trunk revision 829289.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423304/patch-6334.txt against trunk revision 829289. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/105/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423310/patch-6334-1.txt
          against trunk revision 829289.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423310/patch-6334-1.txt against trunk revision 829289. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/19/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The change in the Path constructor may not work since file/directory names may contain #. If we add an assumption that there is no # in file/directory names, then this is an incompatible change.

          I think this problem is again due to our ill-defined path syntax. See also HDFS-13.

          Show
          Tsz Wo Nicholas Sze added a comment - The change in the Path constructor may not work since file/directory names may contain #. If we add an assumption that there is no # in file/directory names, then this is an incompatible change. I think this problem is again due to our ill-defined path syntax. See also HDFS-13 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Also, what is the correct way to handle a path with two or more #, e.g. hdfs://namendoe/a#b/c#d#e ?

          Show
          Tsz Wo Nicholas Sze added a comment - Also, what is the correct way to handle a path with two or more #, e.g. hdfs://namendoe/a#b/c#d#e ?
          Hide
          Amareshwari Sriramadasu added a comment -

          If we add an assumption that there is no # in file/directory names, then this is an incompatible change.

          If Path allows # in the name. And it is known that it does not support fragment in URI.
          i.e.
          URI uri = new URI("file:///foo/bar#baz");
          new Path(uri).toString returns only "file:///foo/bar".
          Then GPO can append fragment portion of the uri, by itself.

          Show
          Amareshwari Sriramadasu added a comment - If we add an assumption that there is no # in file/directory names, then this is an incompatible change. If Path allows # in the name. And it is known that it does not support fragment in URI. i.e. URI uri = new URI("file:///foo/bar#baz"); new Path(uri).toString returns only "file:///foo/bar". Then GPO can append fragment portion of the uri, by itself.
          Hide
          Amareshwari Sriramadasu added a comment -

          Attaching a patch reverting the changes from Path. Modified GenericOptionParser to append fragment of the uri to the final path, if there is any fragment.

          Show
          Amareshwari Sriramadasu added a comment - Attaching a patch reverting the changes from Path. Modified GenericOptionParser to append fragment of the uri to the final path, if there is any fragment.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423433/patch-6334-2.txt
          against trunk revision 829289.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423433/patch-6334-2.txt against trunk revision 829289. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/108/console This message is automatically generated.
          Hide
          Jothi Padmanabhan added a comment -

          This is one possible approach. But I am thinking we could follow another approach.

          Have the Path methods (makeQualified, toString) handle fragments as in patch-6334-1.txt, but do not modify the constructors.
          Modify GOP to construct a URI out of the String as in patch-6334-2.txt and use it to construct the Path with the Path(URI) constructor.

          Since the Path(URI) constructor directly sets the underlying URI and if the Path methods recognize fragments, we should be fine and will not have problems of backward compatibility, no?

          Show
          Jothi Padmanabhan added a comment - This is one possible approach. But I am thinking we could follow another approach. Have the Path methods (makeQualified, toString) handle fragments as in patch-6334-1.txt, but do not modify the constructors. Modify GOP to construct a URI out of the String as in patch-6334-2.txt and use it to construct the Path with the Path(URI) constructor. Since the Path(URI) constructor directly sets the underlying URI and if the Path methods recognize fragments, we should be fine and will not have problems of backward compatibility, no?
          Hide
          Amareshwari Sriramadasu added a comment -

          Modified the patch, as suggested by Jothi

          Show
          Amareshwari Sriramadasu added a comment - Modified the patch, as suggested by Jothi
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423548/patch-6334-3.txt
          against trunk revision 829289.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423548/patch-6334-3.txt against trunk revision 829289. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/112/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch-6334-3.txt looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch-6334-3.txt looks good.
          Hide
          Amareshwari Sriramadasu added a comment -

          Fixed two minor bugs in GenericOptionsParser. GPO is doing an fs.close(), which stops the applications from using FileSystem anymore. Fixed getLibjars method to understand uri on hdfs. Testcases for the same will be added in MAPREDUCE-787.

          Show
          Amareshwari Sriramadasu added a comment - Fixed two minor bugs in GenericOptionsParser. GPO is doing an fs.close(), which stops the applications from using FileSystem anymore. Fixed getLibjars method to understand uri on hdfs. Testcases for the same will be added in MAPREDUCE-787 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423656/patch-6334-4.txt
          against trunk revision 831070.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423656/patch-6334-4.txt against trunk revision 831070. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/118/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch applies to branch 0.21. And all tests passed on branch 0.21 also.

          Show
          Amareshwari Sriramadasu added a comment - Patch applies to branch 0.21. And all tests passed on branch 0.21 also.
          Hide
          Amareshwari Sriramadasu added a comment -

          test-patch passed on branch 0.21

          Show
          Amareshwari Sriramadasu added a comment - test-patch passed on branch 0.21
          Hide
          Jothi Padmanabhan added a comment -

          +1. Patch looks good.

          Show
          Jothi Padmanabhan added a comment - +1. Patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this to 0.21 and above. Thanks, Amareshwari!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this to 0.21 and above. Thanks, Amareshwari!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #75 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/75/)
          . Fix GenericOptionsParser and Path to have a better URI support. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #75 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/75/ ) . Fix GenericOptionsParser and Path to have a better URI support. Contributed by Amareshwari Sriramadasu
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #144 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/144/)
          . Fix GenericOptionsParser and Path to have a better URI support. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #144 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/144/ ) . Fix GenericOptionsParser and Path to have a better URI support. Contributed by Amareshwari Sriramadasu
          Hide
          Amareshwari Sriramadasu added a comment -

          I ran ant test in MAPREDUCE project, picking up the jars from tar ball @ http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/75/. All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124)
          Can somebody commit the core jars to MAPREDUCE project ?
          Core jars are not present in HDFS project after HADOOP-5107.

          Show
          Amareshwari Sriramadasu added a comment - I ran ant test in MAPREDUCE project, picking up the jars from tar ball @ http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/75/ . All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124 ) Can somebody commit the core jars to MAPREDUCE project ? Core jars are not present in HDFS project after HADOOP-5107 .
          Hide
          Sharad Agarwal added a comment -

          core jars updated in MAPREDUCE.

          Show
          Sharad Agarwal added a comment - core jars updated in MAPREDUCE.
          Hide
          Amareshwari Sriramadasu added a comment -

          I picked up the common jars from http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Common-21-Build/lastSuccessfulBuild/artifact/trunk/build/hadoop-core-0.21.0.tar.gz for branch 0.21 and ran ant test in MAPREDUCE project, All tests passed except TestGridmixSubmission (due to MAPREDUE-1124).

          Show
          Amareshwari Sriramadasu added a comment - I picked up the common jars from http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Common-21-Build/lastSuccessfulBuild/artifact/trunk/build/hadoop-core-0.21.0.tar.gz for branch 0.21 and ran ant test in MAPREDUCE project, All tests passed except TestGridmixSubmission (due to MAPREDUE-1124).

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development