Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3951

Tasks are not evenly spread throughout cluster in MR2

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.23.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:
      None

      Description

      In MR1 (at least with the fair and fifo schedulers), if you submit a job that needs fewer resources than the cluster can provide, the tasks are spread relatively evenly across the node. For example, submitting a 100-map job to a 50-node cluster, each with 10 slots, results in 2 tasks on each machine. In MR2, however, the tasks would pile up on the first 10 nodes of the cluster, leaving the other nodes unused. This is highly suboptimal for many use cases.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: