[YARN-10287] Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: pull-request-available
• Assignee: Ashutosh Gupta
• Reporter: Akhil PB
• Created: 22/May/20 13:34
• Updated: 02/Jul/22 16:30
• Resolved: 02/Jul/22 16:30

Description

Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping. The deletion is failed with below error message but the queue got removed from CS configuration and job submission failed and not removed from the backend ZKConfigurationStore. On subsequent modify using scheduler-conf, the queue appears again from ZKConfigurationStore

2020-05-22 12:38:38,252 ERROR org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception thrown when modifying configuration.
java.io.IOException: Failed to re-init queues : mapping contains invalid or non-leaf queue Prod
	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:478)
	at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:430)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2389)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2377)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.updateSchedulerConfiguration(RMWebServices.java:2377)

Repro:

1. Setup Queue Mapping

yarn.scheduler.capacity.root.queues=default,dummy
yarn.scheduler.capacity.queue-mappings=g:hadoop:dummy

2. Stop the root.dummy queue

<update-queue>
       <queue-name>root.dummy</queue-name>
       <params>
         <entry>
           <key>state</key>
           <value>STOPPED</value>
         </entry>
       </params>
     </update-queue>
	 
	 
3. Delete the root.dummy queue

curl --negotiate -u : -X PUT -d @abc.xml -H "Content-type: application/xml" 'http://<RM_IP>:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'

<sched-conf>
      <update-queue>
          <queue-name>root.default</queue-name>
          <params>
            <entry>
              <key>capacity</key>
              <value>100</value>
            </entry>
          </params>
        </update-queue>

        <remove-queue>root.dummy</remove-queue>
      </sched-conf>  

Issue Links

• links to: GitHub Pull Request #4515