Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4018

limit memory usage in jobtracker

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • Reviewed
    • Hide
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Show
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

    Description

      We have seen instances when a user submitted a job with many thousands of mappers. The JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to be restarted.

      One simple proposal would be to limit the maximum number of tasks per job. This can be a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

      Attachments

        1. maxSplits10.patch
          6 kB
          Dhruba Borthakur
        2. maxSplits9.patch
          6 kB
          Dhruba Borthakur
        3. maxSplits8.patch
          7 kB
          Dhruba Borthakur
        4. maxSplits7.patch
          15 kB
          Dhruba Borthakur
        5. maxSplits6.patch
          5 kB
          Dhruba Borthakur
        6. maxSplits5.patch
          4 kB
          Dhruba Borthakur
        7. maxSplits4.patch
          4 kB
          Dhruba Borthakur
        8. maxSplits3.patch
          4 kB
          Dhruba Borthakur
        9. maxSplits2.patch
          2 kB
          Dhruba Borthakur
        10. maxSplits.patch
          2 kB
          Dhruba Borthakur

        Issue Links

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: