Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2214

FairScheduler: preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.5.0
    • 2.6.0
    • scheduler
    • None
    • Reviewed

    Description

      preemptContainerPreCheck() in FSParentQueue rejects preemption requests if the parent queue is below fair share. This can cause a delay in converging towards fairness when the starved leaf queue and the queue above fairshare belong under a non-root parent queue(ie their least common ancestor is a parent queue which is not root).
      Here is an example :
      root.parent has fair share = 80% and usage = 80%
      root.parent.child1 has fair share =40% usage = 80%
      root.parent.child2 has fair share=40% usage=0%

      Now a job is submitted to child2 and the demand is 40%.
      Preemption will kick in and try to reclaim all the 40% from child1.
      When it preempts the first container from child1,the usage of root.parent will become <80%, which is less than root.parent's fair share,causing preemption to stop.So only one container gets preempted in this round although the need is a lot more. child2 would eventually get to half its fair share but only after multiple rounds of preemption.

      Solution is to remove preemptContainerPreCheck() in FSParentQueue and keep it only in FSLeafQueue(which is already there).

      Attachments

        1. YARN-2214-v2.txt
          8 kB
          Ashwin Shankar
        2. YARN-2214-v1.txt
          8 kB
          Ashwin Shankar

        Activity

          People

            ashwinshankar77 Ashwin Shankar
            ashwinshankar77 Ashwin Shankar
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: