Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4613

Scheduling of reduce tasks results in starvation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.23.1, 2.0.1-alpha
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None
    • Environment:

      16 (duo core) machine cluster ==> 32 containers
      namenode and resourcemanager running on separate 17th machine

      Description

      If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.

      I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.

      A work-around is to set the number of reducers < available containers.

        Attachments

        1. scheduling.png
          23 kB
          Vasco

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vasco Vasco
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: