XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:

      Description

      FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.

      Error starting ResourceManager
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:112)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.<init>(AbstractManagedParentQueue.java:51)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.<init>(ManagedParentQueue.java:56)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
      	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
      	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
      	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) 
      

      Input fair-scheduler.xml:

      [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml 
      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
      <allocations>
          <queue name="root">
              <schedulingPolicy>fair</schedulingPolicy>
              <queue name="default">
                  <schedulingPolicy>fair</schedulingPolicy>
              </queue>
              <queue name="users" type="parent">
                  <schedulingPolicy>fair</schedulingPolicy>
              </queue>
          </queue>
          <queuePlacementPolicy>
              <rule name="specified" create="true"/>
              <rule name="nestedUserQueue">
                  <rule name="default" queue="users"/>
              </rule>
              <rule name="default" create="true"/>
          </queuePlacementPolicy>
      </allocations>
      

      Command Used:

       yarn fs2cs -t  -f /tmp/fair-scheduler.xml -y /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o /tmp/CS
      

      Output capacity-scheduler.xml

      <?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
      <property><name>yarn.scheduler.capacity.root.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.users.capacity</name><value>50.000</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.queues</name><value>default,users</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.queue-mappings-override.enable</name><value>false</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.default.capacity</name><value>50.000</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.default.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.maximum-am-resource-percent</name><value>0.5</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.users.auto-create-child-queue.enabled</name><value>true</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.default.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.queue-mappings</name><value>u:%user:%user;u:%user:root.users.%user;u:%user:root.default</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.users.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property>
      <property><name>yarn.scheduler.capacity.root.ordering-policy</name><value>fair</value><final>false</final><source>programmatically</source></property>
      </configuration>
      

      Root Queue is set with App Ordering Policy fair which is wrong

      <property><name>yarn.scheduler.capacity.root.ordering-policy</name><value>fair</value></property>
      

        Attachments

        1. YARN-10127-001.patch
          5 kB
          Peter Bacsko

          Activity

            People

            • Assignee:
              pbacsko Peter Bacsko
              Reporter:
              prabhujoseph Prabhu Joseph
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: