| Property | Description |
|---|---|
yarn.scheduler.capacity.<queue-path>.ordering-policy |
The ordering-policy of the parent queue. Can be one of utilization or priority-utilization. (see notes) |
yarn.scheduler.capacity.<queue-path>.priority |
The priority of queue, higher means more important. It should be a valid integer value, 0 is default value |
yarn.scheduler.capacity.<queue-path>.priority |
The priority of queue, higher means more important. It should be a valid integer value, 0 is default value |
| Property | Description |
|---|---|
yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled |
Do we allow under-utilized queue with higher priority to preempt queue with lower priority even if queue with lower priority is not satisfied. For example, two queues, a and b. A’s priority = 1, (a.used-capacity - a.reserved-capacity) = 40%. B’s priority = 0, b.used-capacity = 30%. Set this configuration to true to allow queue-a to preempt container from queue-b. (The reason why deduct reserved-capacity from used-capacity for queue with higher priority is: the reserved-capacity is just scheduler’s internal implementation to allocate large containers, it is not possible for application to use such reserved-capacity. It is possible that a queue with large container requests have a large number of containers but cannot allocate from any of them. But scheduler will make sure a satisfied queue will not preempt resource from any other queues. A queue is considered to be satisfied when queue’s used-capacity - reserved-capacity >= guaranteed-capacity.) |
yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.reserved-container-delay-ms |
When a reserved container of an underutilized queue is created. Preemption will kick in after specified delay (in ms). The total time to preempt resources for a reserved container from higher priority queue will be: reserved-container-delay-ms + PREEMPTION_WAIT_TIME_BEFORE_KILL. This parameter is added to make preemption from lower priority queue which is underutilized to be more careful. This parameter takes effect when underutilized-preemption.enabled set to true. |
yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.reserved-container-delay-ms |
When doing preemption from under-satisfied queues for priority queue. Do we allow move reserved container from one host to another? (Experimental config, by default it is false) |
priority-utilization ordering policyFor two queues under same parent with the same priority:
For two queues under same parent with different priorities:
utilization ordering policyFor any of two queues under the same parent, the queue with the lowest used-capacity percentage gets the resources.
<configuration>
<property>
<name>yarn.scheduler.capacity.root.ordering-policy</name>
<value>priority-utilization</value>
</property>
<property>
<name>yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.allow-move-reservation</name>
<value>false</value>
</property>
<property>
<name>yarn.scheduler.capacity.ordering-policy.priority-utilization.underutilized-preemption.reserved-container-delay-ms</name>
<value>5000</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>a,b,c,d</value>
<description>The queues at the this level (root is the root queue).</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.capacity</name>
<value>5</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.priority</name>
<value>2</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.user-limit-factor</name>
<value>10</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.capacity</name>
<value>10</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.priority</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.b.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.a.user-limit-factor</name>
<value>5</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.c.capacity</name>
<value>25</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.c.priority</name>
<value>1</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.c.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.c.user-limit-factor</name>
<value>3</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.d.capacity</name>
<value>60</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.d.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.d.user-limit-factor</name>
<value>2</value>
</property>
</configuration>