Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-967

TaskTracker does not need to fully unjar job jars

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • tasktracker
    • None
    • Incompatible change, Reviewed
    • Hide
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.
      Show
      For efficiency, TaskTrackers no longer unjar the job jar into the job cache directory. Users who previously depended on this functionality for shipping non-code dependencies can use the undocumented configuration parameter "mapreduce.job.jar.unpack.pattern" to cause specific jar contents to be unpacked.

    Description

      In practice we have seen some users submitting job jars that consist of 10,000+ classes. Unpacking these jars into mapred.local.dir and then cleaning up after them has a significant cost (both in wall clock and in unnecessary heavy disk utilization). This cost can be easily avoided

      Attachments

        1. mapreduce-967-branch-0.20.txt
          4 kB
          Todd Lipcon
        2. mapreduce-967.txt
          20 kB
          Todd Lipcon
        3. mapreduce-967.txt
          21 kB
          Todd Lipcon
        4. mapreduce-967.txt
          12 kB
          Todd Lipcon
        5. mapreduce-967.txt
          13 kB
          Todd Lipcon
        6. mapreduce-967.txt
          13 kB
          Todd Lipcon
        7. mapreduce-967.txt
          13 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: