Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4740

only .jars can be added to the Distributed Cache classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Koji tracked down this one.

      $ cat mycat.sh
      #!/bin/sh
      cat >& /dev/null
      $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
      export | grep CLASSPATH
      ls -l 
      
      $ myfile=/user/me/myclasses.zip; yarn jar
      hadoop-streaming.jar
      -Dmapreduce.job.cache.archives=hdfs:///${myfile}
      -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh  -file mycat.sh
      

      So, cachearchive like class.zip or class.tar.gz were never set as part of the
      classpath even though they were properly set by DistributedCache.addArchiveToClassPath.

      It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.

      1. MR-4740.txt
        9 kB
        Robert Joseph Evans
      2. MR-4740.txt
        9 kB
        Robert Joseph Evans
      3. MR-4740.txt
        9 kB
        Robert Joseph Evans
      4. MR-4740.txt
        6 kB
        Robert Joseph Evans

        Activity

        Hide
        Robert Joseph Evans added a comment -

        Attaching patch that should fix the issue. It will not work if someone adds an archive with a # in it, and also adds it to the classpath. I do not believe that this works in 1.0 either but i will test it on 1.0 to be sure.

        Show
        Robert Joseph Evans added a comment - Attaching patch that should fix the issue. It will not work if someone adds an archive with a # in it, and also adds it to the classpath. I do not believe that this works in 1.0 either but i will test it on 1.0 to be sure.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550298/MR-4740.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550298/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        OK so the # does work on 1.0.2 if you set the confs manually. I will update the patch accordingly.

        Show
        Robert Joseph Evans added a comment - OK so the # does work on 1.0.2 if you set the confs manually. I will update the patch accordingly.
        Hide
        Robert Joseph Evans added a comment -

        Updated version that allows for a cache archive to have a symlink name, and it to also be added to the class path. This does not update the API to set this that will still not work if there is a #symlink in the archive being added to the classpath, but internally if the confs are set manually it will work, and the unit test verifies it.

        Show
        Robert Joseph Evans added a comment - Updated version that allows for a cache archive to have a symlink name, and it to also be added to the class path. This does not update the API to set this that will still not work if there is a #symlink in the archive being added to the classpath, but internally if the confs are set manually it will work, and the unit test verifies it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550309/MR-4740.txt
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2956//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550309/MR-4740.txt against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2956//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        Sorry looks like 0.23 and trunk are different enough in the MRApps test file that the patch did not apply cleanly. This new version is based off of trunk.

        Show
        Robert Joseph Evans added a comment - Sorry looks like 0.23 and trunk are different enough in the MRApps test file that the patch did not apply cleanly. This new version is based off of trunk.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550314/MR-4740.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550314/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        I missed that files can be named .JAR too. So I made the test case insensitive.

        Show
        Robert Joseph Evans added a comment - I missed that files can be named .JAR too. So I made the test case insensitive.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12550335/MR-4740.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550335/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        +1, lgtm. I'll commit this shortly.

        Show
        Jason Lowe added a comment - +1, lgtm. I'll commit this shortly.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #2908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2908/)
        MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #2908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2908/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Hide
        Jason Lowe added a comment -

        Thanks, Bobby. I committed this to trunk, branch-2, and branch-0.23.

        Show
        Jason Lowe added a comment - Thanks, Bobby. I committed this to trunk, branch-2, and branch-0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #12 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/12/)
        MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

        Result = FAILURE
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #12 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/12/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #413 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/413/)
        svn merge -c 1401054 FIXES: MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401059)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401059
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #413 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/413/ ) svn merge -c 1401054 FIXES: MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401059) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401059 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1204/)
        MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1204/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1234/)
        MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1234/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development