Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Show
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Description

      We have seen instances when a user submitted a job with many thousands of mappers. The JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to be restarted.

      One simple proposal would be to limit the maximum number of tasks per job. This can be a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

      1. maxSplits.patch
        2 kB
        dhruba borthakur
      2. maxSplits2.patch
        2 kB
        dhruba borthakur
      3. maxSplits3.patch
        4 kB
        dhruba borthakur
      4. maxSplits4.patch
        4 kB
        dhruba borthakur
      5. maxSplits5.patch
        4 kB
        dhruba borthakur
      6. maxSplits6.patch
        5 kB
        dhruba borthakur
      7. maxSplits7.patch
        15 kB
        dhruba borthakur
      8. maxSplits8.patch
        7 kB
        dhruba borthakur
      9. maxSplits9.patch
        6 kB
        dhruba borthakur
      10. maxSplits10.patch
        6 kB
        dhruba borthakur

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development