[YARN-11533] CapacityScheduler CapacityConfigType changed in legacy queue allocation mode

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Benjamin Teke
• Created: 17/Jul/23 19:02
• Updated: 20/Jul/23 04:32
• Resolved: 20/Jul/23 04:32

Description

YARN-11000 changed the CapacityConfigType determination method in legacy queue mode, which has some undesired effects, like marking root to be in percentage mode, when the rest of the queue structure is in absolute.

Config:

<configuration>
	<property>
		<name>yarn.scheduler.capacity.root.maximum-capacity</name>
		<value>[memory=40960,vcores=32]</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.root.queues</name>
		<value>default</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.root.capacity</name>
		<value>[memory=40960,vcores=32]</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.schedule-asynchronously.enable</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
		<value>[memory=40960,vcores=32]</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.root.default.maximum-am-resource-percent</name>
		<value>0.2</value>
	</property>
	<property>
		<name>yarn.scheduler.capacity.root.default.capacity</name>
		<value>[memory=40960,vcores=32]</value>
	</property>
</configuration>

Old response:

{
    "scheduler": {
        "schedulerInfo": {
            "type": "capacityScheduler",
            "capacity": 100.0,
            "usedCapacity": 0.0,
            "maxCapacity": 100.0,
            "weight": -1.0,
            "normalizedWeight": 0.0,
            "queueName": "root",
            "queuePath": "root",
            "maxParallelApps": 2147483647,
            "isAbsoluteResource": true,
            "queues": {...}
            "queuePriority": 0,
            "orderingPolicyInfo": "utilization",
            "mode": "absolute",
            "queueType": "parent",
            "creationMethod": "static",
            "autoCreationEligibility": "off",
            "autoQueueTemplateProperties": {},
            "autoQueueParentTemplateProperties": {},
            "autoQueueLeafTemplateProperties": {}
        }
    }
}

New response:

{
    "scheduler": {
        "schedulerInfo": {
            "type": "capacityScheduler",
            "capacity": 100.0,
            "usedCapacity": 0.0,
            "maxCapacity": 100.0,
            "weight": -1.0,
            "normalizedWeight": 0.0,
            "queueName": "root",
            "queuePath": "root",
            "maxParallelApps": 2147483647,
            "isAbsoluteResource": false,
            "queues": {...}
            "queuePriority": 0,
            "orderingPolicyInfo": "utilization",
            "mode": "percentage",
            "queueType": "parent",
            "creationMethod": "static",
            "autoCreationEligibility": "off",
            "autoQueueTemplateProperties": {},
            "autoQueueParentTemplateProperties": {},
            "autoQueueLeafTemplateProperties": {}
        }
    }
}

This is misleading and has some side-effects on various clients.

Issue Links

• links to: GitHub Pull Request #5852
• links to: PR