Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1198 Capacity Scheduler headroom calculation does not work as expected
  3. YARN-2008

CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      If there are two queues, both allowed to use 100% of the actual resources in the cluster. Q1 and Q2 currently use 50% of actual cluster's resources and there is not actual space available. If we use current method to get headroom, CapacityScheduler thinks there are still available resources for users in Q1 but they have been used by Q2.
      If the CapacityScheduelr has a hierarchy queue structure, it may report incorrect queueMaxCap. Here is a example

        rootQueue  
        / \
      L1ParentQueue1   L1ParentQueue2
      (allowed to use up 80% of its parent)   (allowed to use 20% in minimum of its parent)
      / \  
      L2LeafQueue1 L2LeafQueue2  
      (50% of its parent) (50% of its parent in minimum)  

      When we calculate headroom of a user in L2LeafQueue2, current method will think L2LeafQueue2 can use 40% (80%*50%) of actual rootQueue resources. However, without checking L1ParentQueue1, we are not sure. It is possible that L1ParentQueue2 have used 40% of rootQueue resources right now. Actually, L2LeafQueue2 can only use 30% (60%*50%).

        Attachments

        1. YARN-2008.1.patch
          5 kB
          Craig Welch
        2. YARN-2008.2.patch
          13 kB
          Craig Welch
        3. YARN-2008.3.patch
          18 kB
          Craig Welch
        4. YARN-2008.4.patch
          18 kB
          Craig Welch
        5. YARN-2008.5.patch
          19 kB
          Craig Welch
        6. YARN-2008.6.patch
          19 kB
          Craig Welch
        7. YARN-2008.7.patch
          19 kB
          Craig Welch
        8. YARN-2008.8.patch
          20 kB
          Craig Welch
        9. YARN-2008.9.patch
          21 kB
          Craig Welch

          Issue Links

            Activity

              People

              • Assignee:
                cwelch Craig Welch
                Reporter:
                airbots Chen He
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: