[YARN-3627] Preemption not triggered in Fair scheduler when maxResources is set on parent queue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: fairscheduler, scheduler

Environment

Suse 11 SP3, 2 NM

• Assignee: Unassigned
• Reporter: Bibin Chundatt
• Created: 12/May/15 05:00
• Updated: 27/May/15 08:34
• Resolved: 27/May/15 08:32

Description

Consider the below scenario of fair configuration

Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
----Q1.1 (maxResources 4gb)
----Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)

No applications are running in Q2

Submit one application with to Q1.1 with 50 maps & 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.

Preemption will never get triggered since yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the cluster utilization is below .8.

Fairscheduler.java

  private boolean shouldAttemptPreemption() {
    if (preemptionEnabled) {
      return (preemptionUtilizationThreshold < Math.max(
          (float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
          (float) rootMetrics.getAllocatedVirtualCores() /
              clusterResource.getVirtualCores()));
    }
    return false;
  }

Are we supposed to configure in running cluster maxResources <0mb and 0 cores > so that all queues can take full cluster resources always if available??