Major Description
FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
Error starting ResourceManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:112) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.<init>(AbstractManagedParentQueue.java:51) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.<init>(ManagedParentQueue.java:56) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
Input fair-scheduler.xml:
[yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <allocations> <queue name="root"> <schedulingPolicy>fair</schedulingPolicy> <queue name="default"> <schedulingPolicy>fair</schedulingPolicy> </queue> <queue name="users" type="parent"> <schedulingPolicy>fair</schedulingPolicy> </queue> </queue> <queuePlacementPolicy> <rule name="specified" create="true"/> <rule name="nestedUserQueue"> <rule name="default" queue="users"/> </rule> <rule name="default" create="true"/> </queuePlacementPolicy> </allocations>
Command Used:
yarn fs2cs -t -f /tmp/fair-scheduler.xml -y /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o /tmp/CS
Output capacity-scheduler.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration> <property><name>yarn.scheduler.capacity.root.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.users.capacity</name><value>50.000</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.queues</name><value>default,users</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.queue-mappings-override.enable</name><value>false</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.default.capacity</name><value>50.000</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.default.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.maximum-am-resource-percent</name><value>0.5</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.users.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.default.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.queue-mappings</name><value>u:%user:%user;u:%user:root.users.%user;u:%user:root.default</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.users.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property> <property><name>yarn.scheduler.capacity.root.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property> </configuration>
Root Queue is set with App Ordering Policy fair which is wrong
<property><name>yarn.scheduler.capacity.root.ordering-policy</name><value>fair</value></property>