[YARN-9992] Max allocation per queue is zero for custom resource types on RM startup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Target Version/s: 2.10.1
• Assignee: Jonathan Hung
• Reporter: Jonathan Hung
• Created: 26/Nov/19 23:42
• Updated: 05/Dec/19 16:09
• Resolved: 03/Dec/19 21:58

Description

Found an issue where trying to request GPUs on a newly booted RM cannot schedule. It throws the exception in SchedulerUtils#throwInvalidResourceException:

throw new InvalidResourceRequestException(
    "Invalid resource request, requested resource type=[" + reqResourceName
        + "] < 0 or greater than maximum allowed allocation. Requested "
        + "resource=" + reqResource + ", maximum allowed allocation="
        + availableResource
        + ", please note that maximum allowed allocation is calculated "
        + "by scheduler based on maximum resource of registered "
        + "NodeManagers, which might be less than configured "
        + "maximum allocation="
        + ResourceUtils.getResourceTypesMaximumAllocation());

Upon refreshing scheduler (e.g. via refreshQueues), GPU scheduling works again.

I think the RC is that upon scheduler refresh, resource-types.xml is loaded in CapacitySchedulerConfiguration (as part of YARN-7738), so when we call ResourceUtils#fetchMaximumAllocationFromConfig in CapacitySchedulerConfiguration#getMaximumAllocationPerQueue, it's able to fetch the yarn.resource-types config. But resource-types.xml is not loaded into the conf in CapacityScheduler#initScheduler, so it doesn't find the custom resource when computing max allocations, and the custom resource max allocation is 0.

Issue Links

• duplicates: YARN-9205 When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION)