Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2729

Reducers are always counted having "pending tasks" even if they can't be scheduled yet because not enough of their mappers have completed

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.205.0
    • 0.20.205.0
    • None
    • None
    • 0.20.1xx-Secondary

    Description

      In capacity scheduler, number of users in a queue needing slots are calculated based on whether users' jobs have any pending tasks.
      This works fine for map tasks. However, for reduce tasks, jobs do not need reduce slots until the minimum number of map tasks have been completed.

      Here, we add checking whether reduce is ready to schedule (i.e. if a job has completed enough map tasks) when we increment number of users in a queue needing reduce slots.

      Attachments

        1. MAPREDUCE-2729.patch
          10 kB
          Sherry Chen

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sherri_chen Sherry Chen
            sherri_chen Sherry Chen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment