Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2729

Reducers are always counted having "pending tasks" even if they can't be scheduled yet because not enough of their mappers have completed

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      0.20.1xx-Secondary

      Description

      In capacity scheduler, number of users in a queue needing slots are calculated based on whether users' jobs have any pending tasks.
      This works fine for map tasks. However, for reduce tasks, jobs do not need reduce slots until the minimum number of map tasks have been completed.

      Here, we add checking whether reduce is ready to schedule (i.e. if a job has completed enough map tasks) when we increment number of users in a queue needing reduce slots.

        Attachments

          Activity

            People

            • Assignee:
              sherri_chen Sherry Chen
              Reporter:
              sherri_chen Sherry Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: