Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6622

Add capability to set JHS job cache to a task-based limit

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property:
      1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.
      2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.
      Show
      Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: 1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. 2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.

      Description

      When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs can be of varying size. This is generally not a problem when the jobs sizes are uniform or small, but when the job sizes can be very large (say greater than 250k tasks), then the JHS heap size can grow tremendously.

      In cases, where multiple jobs are very large, then the JHS can lock up and spend all its time in GC. However, since the cache is holding on to all the jobs, not much heap space can be freed up.

      By setting a property that sets a cap on the number of tasks allowed in the cache and since the total number of tasks loaded is directly proportional to the amount of heap used, this should help prevent the JHS from locking up.

        Attachments

        1. MAPREDUCE-6622.001.patch
          14 kB
          Ray Chiang
        2. MAPREDUCE-6622.002.patch
          15 kB
          Ray Chiang
        3. MAPREDUCE-6622.003.patch
          15 kB
          Ray Chiang
        4. MAPREDUCE-6622.004.patch
          16 kB
          Ray Chiang
        5. MAPREDUCE-6622.005.patch
          17 kB
          Ray Chiang
        6. MAPREDUCE-6622.006.patch
          17 kB
          Ray Chiang
        7. MAPREDUCE-6622.007.patch
          17 kB
          Ray Chiang
        8. MAPREDUCE-6622.008.patch
          17 kB
          Ray Chiang
        9. MAPREDUCE-6622.009.patch
          20 kB
          Ray Chiang
        10. MAPREDUCE-6622.010.patch
          19 kB
          Ray Chiang
        11. MAPREDUCE-6622.011.patch
          20 kB
          Ray Chiang
        12. MAPREDUCE-6622.012.patch
          23 kB
          Ray Chiang
        13. MAPREDUCE-6622.013.patch
          23 kB
          Ray Chiang
        14. MAPREDUCE-6622.014.patch
          23 kB
          Robert Kanter

          Activity

            People

            • Assignee:
              rchiang Ray Chiang
              Reporter:
              rchiang Ray Chiang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: