Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6849

Preemption by container priority can be problematic for MapReduce

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • applicationmaster
    • None

    Description

      A MapReduce job with thousands of reducers and just a couple of maps left to go was running in a preemptable queue. Periodically other queues would get busy and the RM would preempt some resources from the job, but it always picked the job's map tasks first because they use the lowest priority containers. Even though the reducers had a shorter running time, most were spared but the maps were always shot. Since the map tasks ran for a longer time than the preemption period, the job was in a perpetual preemption loop.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. YARN-3054 Preempt policy in FairScheduler may cause mapreduce job never finish

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5269 Preemption of Reducer (and Shuffle) via checkpointing

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              Unassigned Unassigned
              jlowe Jason Darrell Lowe
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: