[YARN-10505] Extend the maximum-capacity property to support Fair Scheduler migration

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: capacity scheduler
• Assignee: Benjamin Teke
• Reporter: Benjamin Teke
• Created: 30/Nov/20 16:48
• Updated: 18/Aug/21 12:05
• Resolved: 18/Aug/21 12:05

Description

Currently Fair Scheduler supports the following 3 kinds of settings:

• Single percentage (relative to parent) i.e. "X%"
• A set of percentages (relative to parent) i.e. "X% cpu, Y% memory"
• Absolute resources i.e. "X mb, Y vcores"

Please note, that the new, recommended format does not support the single percentage mode, only the last 2, like: “vcores=X, memory-mb=Y” or “vcores=X%, memory-mb=Y%” respectively.

Tasks to accomplish:

1.  It is recommended that all three formats are supported for maximum-capacity in CS after introducing weight mode.
2. Also we want to introduce the percentage modes relative to the cluster, not the parent, i.e The property root.users.maximum-capacity will mean one of the following things: 
   1. Either Parent Percentage: maximum capacity relative to its parent. If it’s set to 50, then it means that the capacity is capped with respect to the parent. This can be covered by the current format, no change there.
   2. Or Cluster Percentage: maximum capacity expressed as a percentage of the overall cluster capacity. This case is the new scenario, for example:
      yarn.scheduler.capacity.root.users.max-capacity = c:50%
      yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30%

Issue Links

• relates to: YARN-10886 Cluster based and parent based max capacity in Capacity Scheduler
• relates to: YARN-10887 Investigation: Decouple capacity and max-capacity modes
• relates to: YARN-9936 Support vector of capacity percentages in Capacity Scheduler configuration