Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-967

TaskTracker does not need to fully unjar job jars

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.
      Show
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.

      Description

      In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided

      1. mapreduce-967-branch-0.20.txt
        4 kB
        Todd Lipcon
      2. mapreduce-967.txt
        20 kB
        Todd Lipcon
      3. mapreduce-967.txt
        21 kB
        Todd Lipcon
      4. mapreduce-967.txt
        12 kB
        Todd Lipcon
      5. mapreduce-967.txt
        13 kB
        Todd Lipcon
      6. mapreduce-967.txt
        13 kB
        Todd Lipcon
      7. mapreduce-967.txt
        13 kB
        Todd Lipcon

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development