[HADOOP-1622] Hadoop should provide a way to allow the user to specify jar file(s) the user job depends on

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0

Release Note

This patch allows new command line options for

hadoop jar
which are

hadoop jar -files <comma seperated list of files> -libjars <comma seperated list of jars> -archives <comma seperated list of archives>

-files options allows you to speficy comma seperated list of path which would be present in your current working directory of your task
-libjars option allows you to add jars to the classpaths of the maps and reduces.
-archives allows you to pass archives as arguments that are unzipped/unjarred and a link with name of the jar/zip are created in the current working directory if tasks.

• Assignee: Mahadev Konar
• Reporter: Runping Qi
• Created: 17/Jul/07 18:51
• Updated: 08/Jul/09 16:52
• Resolved: 26/Mar/08 21:08

Description

More likely than not, a user's job may depend on multiple jars.
Right now, when submitting a job through bin/hadoop, there is no way for the user to specify that.
A walk around for that is to re-package all the dependent jars into a new jar or put the dependent jar files in the lib dir of the new jar.
This walk around causes unnecessary inconvenience to the user. Furthermore, if the user does not own the main function
(like the case when the user uses Aggregate, or datajoin, streaming), the user has to re-package those system jar files too.
It is much desired that hadoop provides a clean and simple way for the user to specify a list of dependent jar files at the time
of job submission. Someting like:

bin/hadoop .... --depending_jars j1.jar:j2.jar

Issue Links

• is duplicated by: HADOOP-366 Should be able to specify more than one jar into a JobConf file
• is related to: MAPREDUCE-574 Fix -file option in Streaming to use Distributed Cache