[YARN-8737] Race condition in ParentQueue when reinitializing and sorting child queues in the meanwhile

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 3.3.0, 2.9.3, 3.2.2, 3.1.4
• Component/s: capacityscheduler
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 31/Aug/18 02:50
• Updated: 20/Dec/21 07:13
• Resolved: 20/Dec/21 07:13

Description

Administrator raised a update for queues through REST API, in RM parent queue is refreshing child queues through calling ParentQueue#reinitialize, meanwhile, async-schedule threads is sorting child queues when calling ParentQueue#sortAndGetChildrenAllocationIterator. Race condition may happen and throw exception as follow because TimSort does not handle the concurrent modification of objects it is sorting:

java.lang.IllegalArgumentException: Comparison method violates its general contract!
        at java.util.TimSort.mergeHi(TimSort.java:899)
        at java.util.TimSort.mergeAt(TimSort.java:516)
        at java.util.TimSort.mergeCollapse(TimSort.java:441)
        at java.util.TimSort.sort(TimSort.java:245)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1454)
        at java.util.Collections.sort(Collections.java:175)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:291)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:804)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:817)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:636)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:2494)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:2431)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersOnMultiNodes(CapacityScheduler.java:2588)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:2676)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.scheduleBasedOnNodeLabels(CapacityScheduler.java:927)
        at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:962)

I think we can add read-lock for ParentQueue#sortAndGetChildrenAllocationIterator to solve this problem, the write-lock will be hold when updating child queues in ParentQueue#reinitialize.

Issue Links

• Blocked: YARN-10058 Capacity Scheduler dispatcher hang when async thread crash
• is fixed by: YARN-10178 Global Scheduler async thread crash caused by 'Comparison method violates its general contract