Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Show
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Description

      We have seen instances when a user submitted a job with many thousands of mappers. The JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to be restarted.

      One simple proposal would be to limit the maximum number of tasks per job. This can be a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

        Attachments

        1. maxSplits10.patch
          6 kB
          dhruba borthakur
        2. maxSplits9.patch
          6 kB
          dhruba borthakur
        3. maxSplits8.patch
          7 kB
          dhruba borthakur
        4. maxSplits7.patch
          15 kB
          dhruba borthakur
        5. maxSplits6.patch
          5 kB
          dhruba borthakur
        6. maxSplits5.patch
          4 kB
          dhruba borthakur
        7. maxSplits4.patch
          4 kB
          dhruba borthakur
        8. maxSplits3.patch
          4 kB
          dhruba borthakur
        9. maxSplits2.patch
          2 kB
          dhruba borthakur
        10. maxSplits.patch
          2 kB
          dhruba borthakur

          Issue Links

            Activity

              People

              • Assignee:
                dhruba dhruba borthakur
                Reporter:
                dhruba dhruba borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: