[MAPREDUCE-4740] only .jars can be added to the Distributed Cache classpath

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.2-alpha
• Fix Version/s: 2.0.3-alpha, 0.23.5
• Component/s: mrv2
• Target Version/s: 0.23.5
• Hadoop Flags: Reviewed
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 22/Oct/12 14:57
• Updated: 19/Aug/15 00:16
• Resolved: 22/Oct/12 20:00

Description

Koji tracked down this one.

$ cat mycat.sh
#!/bin/sh
cat >& /dev/null
$JAVA_HOME/bin/jinfo $PPID | grep java.class.path
export | grep CLASSPATH
ls -l 

$ myfile=/user/me/myclasses.zip; yarn jar
hadoop-streaming.jar
-Dmapreduce.job.cache.archives=hdfs:///${myfile}
-Dmapreduce.job.classpath.archives=${myfile} -input in.txt -output out -reducer NONE -mapper mycat.sh  -file mycat.sh

So, cachearchive like class.zip or class.tar.gz were never set as part of the
classpath even though they were properly set by DistributedCache.addArchiveToClassPath.

It looks like we are parsing the classpath out of the configs, but then throwing that away. It looks simple enough to add them in the correct place.

Issue Links

• is related to: MAPREDUCE-6454 MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.