Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Show
      Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

      Description

      We have seen instances when a user submitted a job with many thousands of mappers. The JobTracker was running with 3GB heap, but it was still not enough to prevent memory trashing from Garbage collection; effectively the Job Tracker was not able to serve jobs and had to be restarted.

      One simple proposal would be to limit the maximum number of tasks per job. This can be a configurable parameter. Is there other things that eat huge globs of memory in job Tracker?

      1. maxSplits9.patch
        6 kB
        dhruba borthakur
      2. maxSplits8.patch
        7 kB
        dhruba borthakur
      3. maxSplits7.patch
        15 kB
        dhruba borthakur
      4. maxSplits6.patch
        5 kB
        dhruba borthakur
      5. maxSplits5.patch
        4 kB
        dhruba borthakur
      6. maxSplits4.patch
        4 kB
        dhruba borthakur
      7. maxSplits3.patch
        4 kB
        dhruba borthakur
      8. maxSplits2.patch
        2 kB
        dhruba borthakur
      9. maxSplits10.patch
        6 kB
        dhruba borthakur
      10. maxSplits.patch
        2 kB
        dhruba borthakur

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          dhruba borthakur made changes -
          Link This issue duplicates HADOOP-3925 [ HADOOP-3925 ]
          Robert Chansler made changes -
          Release Note A new configuration parameter named mapred.max.tasks.per.job that specifies the maximum number of tasks per job.

          Introduced new configuration parameter mapred.max.tasks.per.job to specifie the maximum number of tasks per job.

          dhruba borthakur made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          dhruba borthakur made changes -
          Hadoop Flags [Reviewed]
          dhruba borthakur made changes -
          Fix Version/s 0.19.0 [ 12313211 ]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Attachment maxSplits10.patch [ 12391676 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-4261 [ HADOOP-4261 ]
          dhruba borthakur made changes -
          Link This issue blocks HADOOP-4261 [ HADOOP-4261 ]
          dhruba borthakur made changes -
          Link This issue blocks HADOOP-4261 [ HADOOP-4261 ]
          dhruba borthakur made changes -
          Attachment maxSplits9.patch [ 12391403 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Release Note A new configuration parameter named mapred.max.tasks.per.job that specifies the maximum number of tasks per job.

          dhruba borthakur made changes -
          Attachment maxSplits8.patch [ 12390777 ]
          Owen O'Malley made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Attachment maxSplits7.patch [ 12389601 ]
          dhruba borthakur made changes -
          Attachment maxSplits6.patch [ 12389479 ]
          dhruba borthakur made changes -
          Attachment maxSplits5.patch [ 12389412 ]
          dhruba borthakur made changes -
          Attachment maxSplits4.patch [ 12389405 ]
          dhruba borthakur made changes -
          Attachment maxSplits3.patch [ 12389221 ]
          dhruba borthakur made changes -
          Attachment maxSplits2.patch [ 12389153 ]
          dhruba borthakur made changes -
          Field Original Value New Value
          Attachment maxSplits.patch [ 12389134 ]
          dhruba borthakur created issue -

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development