Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-771

Setup and cleanup tasks remain in UNASSIGNED state for a long time on tasktrackers with long running high RAM tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a high RAM job's task is scheduled on a tasktracker, the number of slots it occupies will be more than one. If enough such tasks are scheduled, there can come a situation when there are no more slots free on the node but the number of tasks running is less than the number of slots. The jobtracker currently schedules a setup or cleanup task based on how many tasks are running on the system, rather than slots. As a result of this, it can schedule a setup or cleanup task on a tasktracker without any free slots. If the high RAM job's tasks are long running, this will significantly delay the running of the setup or cleanup task, and thus the entire job.

      1. MAPREDUCE-771.patch
        10 kB
        Hemanth Yamijala
      2. MAPREDUCE-771.patch
        10 kB
        Hemanth Yamijala
      3. MAPREDUCE-771-20.patch
        1 kB
        Hemanth Yamijala

        Activity

        Hemanth Yamijala created issue -
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Fix Version/s 0.21.0 [ 12314045 ]
        Affects Version/s 0.21.0 [ 12314045 ]
        Affects Version/s 0.20.1 [ 12314047 ]
        Hemanth Yamijala made changes -
        Attachment MAPREDUCE-771.patch [ 12413959 ]
        Arun C Murthy made changes -
        Assignee Hemanth Yamijala [ yhemanth ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Hemanth Yamijala made changes -
        Attachment MAPREDUCE-771.patch [ 12413973 ]
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hemanth Yamijala made changes -
        Attachment MAPREDUCE-771-20.patch [ 12413990 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development