[YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.2.2, 3.1.4, 2.10.1
• Fix Version/s: 3.4.0, 3.3.1, 2.10.2, 3.2.3
• Component/s: capacity scheduler, scheduler preemption
• Assignee: Eric Payne
• Reporter: Eric Payne
• Created: 03/Feb/21 20:32
• Updated: 10/Jun/21 08:15
• Resolved: 25/Feb/21 20:32

Description

YARN-8292 added code that prevents CS intra-queue preemption from preempting containers from an app unless all of the major resources used by the app are greater than the user limit for that user.

Ex:

Used	User Limit
<58GB, 58>	<30GB, 300>

In this example, only used memory is above the user limit, not used vcores. So, intra-queue preemption will not occur.

YARN-8292 added the conservativeDRF flag to CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain. If conservativeDRF is false, containers will be preempted from apps in the example state. If true, containers will not be preempted.

This flag is hard-coded to false for Inter-queue (cross-queue) preemption and true for intra-queue (in-queue) preemption.

I propose that in some cases, we want intra-queue preemption to be more aggressive and preempt in the example case. To accommodate that, I propose the addition of a config property.

Also, we may want inter-queue (cross-queue) preemption to be more conservative, so I propose also making that a configuration property: