Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10528

maxAMShare should only be accepted for leaf queues, not parent queues

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0, 3.3.1, 3.1.5, 3.2.3
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      Based on Hadoop documentation, it is clear that maxAMShare property can only be used for leaf queues. This is similar to the reservation setting.

      However, existing code only ensures that the reservation setting is not accepted for "parent" queues (see https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226 and https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233) but it is missing the checks for maxAMShare. Due to this, it is currently possible to have an allocation similar to below:

      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
          <queue name="root">
              <queue name="default">
              <queue name="users" type="parent">
              <rule name="specified" create="true"/>
              <rule name="nestedUserQueue" create="true">
                  <rule name="default" create="true" queue="users"/>
              <rule name="default"/>

      where maxAMShare is 1.0f meaning, it is possible allocate 100% of the queue's resources for Application Masters. Notice above that root.users is a parent queue, however, it still gladly accepts maxAMShare. This is contrary to the documentation and in fact, it is very misleading because the child queues like root.users.<user> actually do not inherit this setting at all and they still go on and use the default of 0.5 instead of 1.0, see the attached screenshot as an example.


        1. maxAMShare for root.users (parent queue) has no effect as child queue does not inherit it.png
          871 kB
          Siddharth Ahuja
        2. YARN-10528.001.patch
          7 kB
          Siddharth Ahuja
        3. YARN-10528.002.patch
          7 kB
          Siddharth Ahuja
        4. YARN-10528-branch-3.1.001.patch
          7 kB
          Siddharth Ahuja
        5. YARN-10528-branch-3.2.001.patch
          7 kB
          Siddharth Ahuja
        6. YARN-10528-branch-3.3.001.patch
          7 kB
          Siddharth Ahuja



            • Assignee:
              sahuja Siddharth Ahuja
              sahuja Siddharth Ahuja


              • Created:

                Issue deployment