Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6849

Preemption by container priority can be problematic for MapReduce

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: applicationmaster
    • Labels:
      None

      Description

      A MapReduce job with thousands of reducers and just a couple of maps left to go was running in a preemptable queue. Periodically other queues would get busy and the RM would preempt some resources from the job, but it always picked the job's map tasks first because they use the lowest priority containers. Even though the reducers had a shorter running time, most were spared but the maps were always shot. Since the map tasks ran for a longer time than the preemption period, the job was in a perpetual preemption loop.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jlowe Jason Darrell Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: