[HADOOP-3570] Including user specified jar files in the client side classpath path in Hadoop 0.17 streaming

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.19.0
• Hadoop Flags: Reviewed
• Assignee: Sharad Agarwal
• Reporter: Suhas
• Created: 16/Jun/08 17:43
• Updated: 08/Jul/09 17:05
• Resolved: 16/Sep/08 12:50

Description

Hadoop 0.17 streaming allows specifying multiple user specific jar files on the command line along with the hadoop-streaming.jar. These user specific jar files are not automatically added on the client side class path when hadoop runs the streaming job

For example, in the following streaming job <user-app.jar> is not automatically included in client side path before executing the streaming job

hadoop jar -libjars <user-app.jar> $HADOOP_HOME/hadoop-streaming.jar \
-input </input-data-path> \
-output </output data path> \
-inputformat <Input-format> \
-outputformat <OutputFormat> \
-mapper /bin/cat -reducer /bin/cat \
-additionalconfspec <xxxx> \
-jobconf mapred.reduce.tasks=3\

Workaround:
Current work around is to set these user specific jar files in
HADOOP_CLASSPATH environment variable on the client side before running above example.