[YARN-10821] User limit is not calculated as per definition for preemption

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: capacity scheduler
• Assignee: Andras Gyori
• Reporter: Andras Gyori
• Created: 14/Jun/21 12:46
• Updated: 10/Feb/22 19:31
• Resolved: 10/Feb/22 19:31

Description

Minimum user limit percent (MULP) is a soft limit by definition. Preemption uses pending resources to determine the resources needed by a queue, which is calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This method involves headroom calculated by UsersManager#computeUserLimit. However, the pending resources for preemption are limited in an unexpected fashion.

• In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is calculated first:

   float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
   1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
  

• In UsersManager#computeUserLimit the userLimit is calculated as is (currentCapacity * userLimit)

   Resource userLimitResource = Resources.max(resourceCalculator,
   partitionResource,
   Resources.divideAndCeil(resourceCalculator, resourceUsed,
   usersSummedByWeight),
   Resources.divideAndCeil(resourceCalculator,
   Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
   100));
  

The fewer users occupying the queue, the more prevalent and outstanding this effect will be in preemption.