Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2729

Reducers are always counted having "pending tasks" even if they can't be scheduled yet because not enough of their mappers have completed

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      0.20.1xx-Secondary

      Description

      In capacity scheduler, number of users in a queue needing slots are calculated based on whether users' jobs have any pending tasks.
      This works fine for map tasks. However, for reduce tasks, jobs do not need reduce slots until the minimum number of map tasks have been completed.

      Here, we add checking whether reduce is ready to schedule (i.e. if a job has completed enough map tasks) when we increment number of users in a queue needing reduce slots.

        Activity

        Sherry Chen created issue -
        Sherry Chen made changes -
        Field Original Value New Value
        Affects Version/s 0.20.205.0 [ 12316391 ]
        Fix Version/s 0.20.205.0 [ 12316391 ]
        Sherry Chen made changes -
        Attachment MAPREDUCE-2729.patch [ 12487910 ]
        Sherry Chen made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sherry Chen
            Reporter:
            Sherry Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development