Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6501

Improve reducer preemption

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.1
    • Fix Version/s: None
    • Component/s: applicationmaster
    • Labels:
      None

      Description

      As discussed on MAPREDUCE-6302, we could improve the reducer preemption as follows:

      1. preempt enough reducers so there are enough mappers to match slowstart
      2. prioritize preempting reducers that are still in SHUFFLE phase
      3. add an option to not preempt reducers that are past SHUFFLE phase irrespective of slowstart as long as one mapper can run

      We could do it all in one patch or create subtasks as necessary.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pbacsko Peter Bacsko
                Reporter:
                kasha Karthik Kambatla
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: