[MAPREDUCE-967] TaskTracker does not need to fully unjar job jars

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0
• Component/s: tasktracker
• Hadoop Flags: Incompatible change, Reviewed

Release Note

For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.

• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 10/Sep/09 06:26
• Updated: 24/Aug/10 21:17
• Resolved: 15/Dec/09 20:56

Description

In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided

Issue Links

• is blocked by: HADOOP-6346 Add support for specifying unpack pattern regex to RunJar.unJar
• relates to: MAPREDUCE-727 Move the bin/hadoop jar command over to bin/mapred