Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4018

limit memory usage in jobtracker

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • Reviewed
    • Hide
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Show
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

    Description

      We have seen instances when a user submitted a job with many thousands of mappers. The JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to be restarted.

      One simple proposal would be to limit the maximum number of tasks per job. This can be a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

      Attachments

        1. maxSplits10.patch
          6 kB
          Dhruba Borthakur
        2. maxSplits9.patch
          6 kB
          Dhruba Borthakur
        3. maxSplits8.patch
          7 kB
          Dhruba Borthakur
        4. maxSplits7.patch
          15 kB
          Dhruba Borthakur
        5. maxSplits6.patch
          5 kB
          Dhruba Borthakur
        6. maxSplits5.patch
          4 kB
          Dhruba Borthakur
        7. maxSplits4.patch
          4 kB
          Dhruba Borthakur
        8. maxSplits3.patch
          4 kB
          Dhruba Borthakur
        9. maxSplits2.patch
          2 kB
          Dhruba Borthakur
        10. maxSplits.patch
          2 kB
          Dhruba Borthakur

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-3925 Configuration paramater to set the maximum number of mappers/reducers for a job

          • Major - Major loss of function.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4261 Jobs failing in the init stage will never cleanup

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: