[YARN-10171] Add support for increment-allocation of custom resource types

• Type: Sub-task
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: yarn
• Assignee: Qi Zhu
• Reporter: Adam Antal
• Created: 27/Feb/20 11:43
• Updated: 06/Jul/21 11:43

Description

The FairScheduler's yarn.resource-types.memory-mb.increment-allocation and yarn.resource-types.vcores.increment-allocation configs are converted to the yarn.scheduler.minimum-allocation-* configs, which is fine for the vcores and memory.

In case of custom resource types like GPU if yarn.resource-types.gpu.increment-allocation is set, then CS will not be aware of that. We don't have a yarn.scheduler.minimum-allocation-gpu setting for this purpose, but yarn.resource-types.gpu.min-allocation is respected by the ResourceCalculator through the ResourceUtils#getResourceInformationMapFromConfig which would provide us with the same behaviour.