Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-787

-files, -archives should honor user given symlink path

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fix JobSubmitter to honor user given symlink in the path.

      Description

      Currently, if user gives an option such as
      -files hdfs://host:fs_port/user/testfile.txt#testlink
      The symlink name "testlink" is not honored. It alwasys creates symlink with name testfile.txt in cwd of the task.

      If the user has given a symlink name, it should be honored. If no symlink-name is given, then the path.getName() can be used.

      1. patch-787-ydist.txt
        26 kB
        Amareshwari Sriramadasu
      2. patch-787-ydist.txt
        25 kB
        Amareshwari Sriramadasu
      3. patch-787-3.txt
        12 kB
        Amareshwari Sriramadasu
      4. patch-787-3.txt
        12 kB
        Amareshwari Sriramadasu
      5. patch-787-2.txt
        12 kB
        Amareshwari Sriramadasu
      6. patch-787-1-ydist.txt
        26 kB
        Amareshwari Sriramadasu
      7. patch-787-1.txt
        10 kB
        Amareshwari Sriramadasu
      8. patch-787.txt
        8 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Amareshwari Sriramadasu added a comment -

          Patch fixes JobSubmitter to understand URI for -files, -archives option.
          Also fixed a minor bug in TaskDistributedCacheManager.

          Updated the testcase TestCommandLineJobSubmission to test the above.

          Show
          Amareshwari Sriramadasu added a comment - Patch fixes JobSubmitter to understand URI for -files, -archives option. Also fixed a minor bug in TaskDistributedCacheManager. Updated the testcase TestCommandLineJobSubmission to test the above.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423810/patch-787.txt
          against trunk revision 831816.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 11 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423810/patch-787.txt against trunk revision 831816. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/119/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Updated forrest documentation

          Show
          Amareshwari Sriramadasu added a comment - Updated forrest documentation
          Hide
          Amareshwari Sriramadasu added a comment -

          Ran ant docs on my machine. It passed successfully.

          Show
          Amareshwari Sriramadasu added a comment - Ran ant docs on my machine. It passed successfully.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12423903/patch-787-1.txt
          against trunk revision 832329.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423903/patch-787-1.txt against trunk revision 832329. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/123/console This message is automatically generated.
          Hide
          Jothi Padmanabhan added a comment -

          This looks good. Some minor points:

          1. In TaskDistributedCacheManager.makeCacheFiles, I think we should compare URI's instead of paths
          2. The documentation in mapred_tutorial can be a little more descriptive
          3. fstream.close is missing for file f1 in TestCommandLineJobSubmission
          Show
          Jothi Padmanabhan added a comment - This looks good. Some minor points: In TaskDistributedCacheManager.makeCacheFiles , I think we should compare URI's instead of paths The documentation in mapred_tutorial can be a little more descriptive fstream.close is missing for file f1 in TestCommandLineJobSubmission
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch incorporates comments 2 and 3.

          In TaskDistributedCacheManager.makeCacheFiles, I think we should compare URI's instead of paths

          This looks like an intrusive change. This would require more public apis to changes. Can be done in different jira.

          Show
          Amareshwari Sriramadasu added a comment - Patch incorporates comments 2 and 3. In TaskDistributedCacheManager.makeCacheFiles, I think we should compare URI's instead of paths This looks like an intrusive change. This would require more public apis to changes. Can be done in different jira.
          Hide
          Amareshwari Sriramadasu added a comment -

          Ran ant docs on my machine. It passed successfully.

          Show
          Amareshwari Sriramadasu added a comment - Ran ant docs on my machine. It passed successfully.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424021/patch-787-2.txt
          against trunk revision 832362.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424021/patch-787-2.txt against trunk revision 832362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/124/console This message is automatically generated.
          Hide
          Jothi Padmanabhan added a comment -

          Minor editorial comments on the doc changes:

          1. I think it is better to say "Users can" instead of "User can"
          2. In streaming.xml, the word "different" is misspelt.
          Show
          Jothi Padmanabhan added a comment - Minor editorial comments on the doc changes: I think it is better to say "Users can" instead of "User can" In streaming.xml, the word "different" is misspelt.
          Hide
          Amareshwari Sriramadasu added a comment -

          Corrected the doc.
          Ran ant docs on my machine. It passed successfully.

          Show
          Amareshwari Sriramadasu added a comment - Corrected the doc. Ran ant docs on my machine. It passed successfully.
          Hide
          Jothi Padmanabhan added a comment -

          This looks like an intrusive change. This would require more public apis to changes. Can be done in different jira.

          sigh We should cleanup this indiscriminate usage of interchanging path and uri some time.
          +1 for the patch.

          Show
          Jothi Padmanabhan added a comment - This looks like an intrusive change. This would require more public apis to changes. Can be done in different jira. sigh We should cleanup this indiscriminate usage of interchanging path and uri some time. +1 for the patch.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          sigh We should cleanup this indiscriminate usage of interchanging path and uri some time.

          MAPREDUCE-950 was already filed for this.

          Show
          Vinod Kumar Vavilapalli added a comment - sigh We should cleanup this indiscriminate usage of interchanging path and uri some time. MAPREDUCE-950 was already filed for this.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for Yahoo! distribution branch 0.20.
          Ran ant test and test-patch. test-patch passed.
          All tests passed, except TestJobHistory (due to HADOOP-5647), TestKillSubProcesses(due to MAPREDUCE-408) and TestReduceFetch (due to MAPREDUCE-433).

          Show
          Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution branch 0.20. Ran ant test and test-patch. test-patch passed. All tests passed, except TestJobHistory (due to HADOOP-5647 ), TestKillSubProcesses(due to MAPREDUCE-408 ) and TestReduceFetch (due to MAPREDUCE-433 ).
          Hide
          Amareshwari Sriramadasu added a comment -

          patch-787-ydist.txt should be committed after HADOOP-4864. Otherwise, new added tests would fail.

          Show
          Amareshwari Sriramadasu added a comment - patch-787-ydist.txt should be committed after HADOOP-4864 . Otherwise, new added tests would fail.
          Hide
          Iyappan Srinivasan added a comment -

          All test secenarios passed.

          -files option:

          without #,with #,filenames as numbers, Special charecters, Multiple files, hdfs file, Multiple files, Combination of hdfs and lfs,Combination of hdfs and lfs without #, negative scenarios like file does not exist

          -libjars : normal , multiple files with libjars and non-existant libjars, hdfs libjars path,

          -archives:
          without #, without # and multiple archives, with # and multiple archives,With numeric tag with special charecters, negative scenario of non-existing archive file

          Show
          Iyappan Srinivasan added a comment - All test secenarios passed. -files option: without #,with #,filenames as numbers, Special charecters, Multiple files, hdfs file, Multiple files, Combination of hdfs and lfs,Combination of hdfs and lfs without #, negative scenarios like file does not exist -libjars : normal , multiple files with libjars and non-existant libjars, hdfs libjars path, -archives: without #, without # and multiple archives, with # and multiple archives,With numeric tag with special charecters, negative scenario of non-existing archive file
          Show
          Amareshwari Sriramadasu added a comment - All tests passed with common jars picked up from http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Common-21-Build/lastSuccessfulBuild/artifact/trunk/build/hadoop-core-0.21.0.tar.gz in branch 0.21.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch-787-ydist.txt looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch-787-ydist.txt looks good.
          Hide
          Amareshwari Sriramadasu added a comment -

          Re-submitting the same patch for hudson, since the common jars will be picked up from maven.

          Show
          Amareshwari Sriramadasu added a comment - Re-submitting the same patch for hudson, since the common jars will be picked up from maven.
          Hide
          Amareshwari Sriramadasu added a comment -

          Re-ran ant test in branch 0.21, with the patch. All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124)

          Show
          Amareshwari Sriramadasu added a comment - Re-ran ant test in branch 0.21, with the patch. All the tests passed except TestGridmixSubmission (due to MAPREDUCE-1124 )
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426262/patch-787-3.txt
          against trunk revision 884628.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426262/patch-787-3.txt against trunk revision 884628. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/152/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Test failure is due to MAPREDUCE-1245

          Show
          Amareshwari Sriramadasu added a comment - Test failure is due to MAPREDUCE-1245
          Hide
          Sharad Agarwal added a comment -

          I just committed this to 0.21 and trunk. Thanks Amareshwari.

          Show
          Sharad Agarwal added a comment - I just committed this to 0.21 and trunk. Thanks Amareshwari.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #162 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/162/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #162 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/162/ )
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for Yahoo! distribution security branch.

          Show
          Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution security branch.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch updates streaming doc, missed in earlier patch.

          Show
          Amareshwari Sriramadasu added a comment - Patch updates streaming doc, missed in earlier patch.
          Hide
          Ravi Gummadi added a comment -

          patch-787-1-ydist.txt looks good.

          Some very minor comments:

          (1) Changes in streaming.xml have ^M
          (2) Indentation in "if (archives != null) {" needs 1 more space at the beginning of all lines inside this if block.
          (3) Though unrelated to this patch, we can remove the unused imports in TaskDistributedCacheManager.java

          Show
          Ravi Gummadi added a comment - patch-787-1-ydist.txt looks good. Some very minor comments: (1) Changes in streaming.xml have ^M (2) Indentation in "if (archives != null) {" needs 1 more space at the beginning of all lines inside this if block. (3) Though unrelated to this patch, we can remove the unused imports in TaskDistributedCacheManager.java
          Hide
          Amareshwari Sriramadasu added a comment -

          Changes in streaming.xml have ^M

          Already existing lines are also showing ^M characters in the patch. They are not introduced in the patch. These characters are not seen after applying the patch.

          Indentation in "if (archives != null) {" needs 1 more space at the beginning of all lines inside this if block.

          I could not find this if block in the patch. I guess you are referring to already existing code.

          So, not generating a new patch.

          Show
          Amareshwari Sriramadasu added a comment - Changes in streaming.xml have ^M Already existing lines are also showing ^M characters in the patch. They are not introduced in the patch. These characters are not seen after applying the patch. Indentation in "if (archives != null) {" needs 1 more space at the beginning of all lines inside this if block. I could not find this if block in the patch. I guess you are referring to already existing code. So, not generating a new patch.
          Hide
          Ravi Gummadi added a comment -

          That is already existing code only.
          +1 for patch-787-1-ydist.tx

          Show
          Ravi Gummadi added a comment - That is already existing code only. +1 for patch-787-1-ydist.tx

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development