Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2214

FairScheduler: preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.6.0
    • Component/s: scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      preemptContainerPreCheck() in FSParentQueue rejects preemption requests if the parent queue is below fair share. This can cause a delay in converging towards fairness when the starved leaf queue and the queue above fairshare belong under a non-root parent queue(ie their least common ancestor is a parent queue which is not root).
      Here is an example :
      root.parent has fair share = 80% and usage = 80%
      root.parent.child1 has fair share =40% usage = 80%
      root.parent.child2 has fair share=40% usage=0%

      Now a job is submitted to child2 and the demand is 40%.
      Preemption will kick in and try to reclaim all the 40% from child1.
      When it preempts the first container from child1,the usage of root.parent will become <80%, which is less than root.parent's fair share,causing preemption to stop.So only one container gets preempted in this round although the need is a lot more. child2 would eventually get to half its fair share but only after multiple rounds of preemption.

      Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it only in FSLeafQueue(which is already there).

        Attachments

        1. YARN-2214-v2.txt
          8 kB
          Ashwin Shankar
        2. YARN-2214-v1.txt
          8 kB
          Ashwin Shankar

          Activity

            People

            • Assignee:
              ashwinshankar77 Ashwin Shankar
              Reporter:
              ashwinshankar77 Ashwin Shankar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: