Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4740

only .jars can be added to the Distributed Cache classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Koji tracked down this one.

      $ cat mycat.sh
      #!/bin/sh
      cat >& /dev/null
      $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
      export | grep CLASSPATH
      ls -l 
      
      $ myfile=/user/me/myclasses.zip; yarn jar
      hadoop-streaming.jar
      -Dmapreduce.job.cache.archives=hdfs:///${myfile}
      -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh  -file mycat.sh
      

      So, cachearchive like class.zip or class.tar.gz were never set as part of the
      classpath even though they were properly set by DistributedCache.addArchiveToClassPath.

      It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.

      1. MR-4740.txt
        9 kB
        Robert Joseph Evans
      2. MR-4740.txt
        9 kB
        Robert Joseph Evans
      3. MR-4740.txt
        9 kB
        Robert Joseph Evans
      4. MR-4740.txt
        6 kB
        Robert Joseph Evans

        Issue Links

          Activity

          Hide
          Robert Joseph Evans added a comment -

          Attaching patch that should fix the issue. It will not work if someone adds an archive with a # in it, and also adds it to the classpath. I do not believe that this works in 1.0 either but i will test it on 1.0 to be sure.

          Show
          Robert Joseph Evans added a comment - Attaching patch that should fix the issue. It will not work if someone adds an archive with a # in it, and also adds it to the classpath. I do not believe that this works in 1.0 either but i will test it on 1.0 to be sure.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550298/MR-4740.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550298/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2955//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          OK so the # does work on 1.0.2 if you set the confs manually. I will update the patch accordingly.

          Show
          Robert Joseph Evans added a comment - OK so the # does work on 1.0.2 if you set the confs manually. I will update the patch accordingly.
          Hide
          Robert Joseph Evans added a comment -

          Updated version that allows for a cache archive to have a symlink name, and it to also be added to the class path. This does not update the API to set this that will still not work if there is a #symlink in the archive being added to the classpath, but internally if the confs are set manually it will work, and the unit test verifies it.

          Show
          Robert Joseph Evans added a comment - Updated version that allows for a cache archive to have a symlink name, and it to also be added to the class path. This does not update the API to set this that will still not work if there is a #symlink in the archive being added to the classpath, but internally if the confs are set manually it will work, and the unit test verifies it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550309/MR-4740.txt
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2956//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550309/MR-4740.txt against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2956//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          Sorry looks like 0.23 and trunk are different enough in the MRApps test file that the patch did not apply cleanly. This new version is based off of trunk.

          Show
          Robert Joseph Evans added a comment - Sorry looks like 0.23 and trunk are different enough in the MRApps test file that the patch did not apply cleanly. This new version is based off of trunk.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550314/MR-4740.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550314/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2957//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          I missed that files can be named .JAR too. So I made the test case insensitive.

          Show
          Robert Joseph Evans added a comment - I missed that files can be named .JAR too. So I made the test case insensitive.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550335/MR-4740.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550335/MR-4740.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2958//console This message is automatically generated.
          Hide
          Jason Lowe added a comment -

          +1, lgtm. I'll commit this shortly.

          Show
          Jason Lowe added a comment - +1, lgtm. I'll commit this shortly.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2908/)
          MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2908/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Hide
          Jason Lowe added a comment -

          Thanks, Bobby. I committed this to trunk, branch-2, and branch-0.23.

          Show
          Jason Lowe added a comment - Thanks, Bobby. I committed this to trunk, branch-2, and branch-0.23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #12 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/12/)
          MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

          Result = FAILURE
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #12 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/12/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #413 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/413/)
          svn merge -c 1401054 FIXES: MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401059)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401059
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #413 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/413/ ) svn merge -c 1401054 FIXES: MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401059) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401059 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1204/)
          MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1204 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1204/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1234/)
          MAPREDUCE-4740. only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054)

          Result = SUCCESS
          jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1234 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1234/ ) MAPREDUCE-4740 . only .jars can be added to the Distributed Cache classpath. Contributed by Robert Joseph Evans (Revision 1401054) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1401054 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java

            People

            • Assignee:
              Robert Joseph Evans
              Reporter:
              Robert Joseph Evans
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development