Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4740

only .jars can be added to the Distributed Cache classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Koji tracked down this one.

      $ cat mycat.sh
      #!/bin/sh
      cat >& /dev/null
      $JAVA_HOME/bin/jinfo $PPID | grep java.class.path
      export | grep CLASSPATH
      ls -l 
      
      $ myfile=/user/me/myclasses.zip; yarn jar
      hadoop-streaming.jar
      -Dmapreduce.job.cache.archives=hdfs:///${myfile}
      -Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh  -file mycat.sh
      

      So, cachearchive like class.zip or class.tar.gz were never set as part of the
      classpath even though they were properly set by DistributedCache.addArchiveToClassPath.

      It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.

      1. MR-4740.txt
        6 kB
        Robert Joseph Evans
      2. MR-4740.txt
        9 kB
        Robert Joseph Evans
      3. MR-4740.txt
        9 kB
        Robert Joseph Evans
      4. MR-4740.txt
        9 kB
        Robert Joseph Evans

        Activity

        Robert Joseph Evans created issue -
        Robert Joseph Evans made changes -
        Field Original Value New Value
        Attachment MR-4740.txt [ 12550298 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Joseph Evans made changes -
        Attachment MR-4740.txt [ 12550309 ]
        Robert Joseph Evans made changes -
        Attachment MR-4740.txt [ 12550314 ]
        Robert Joseph Evans made changes -
        Attachment MR-4740.txt [ 12550335 ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.3-alpha [ 12323275 ]
        Fix Version/s 0.23.5 [ 12323312 ]
        Resolution Fixed [ 1 ]
        Thomas Graves made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development