[YARN-11147] ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: András Győri
• Reporter: András Győri
• Created: 11/May/22 15:40
• Updated: 28/Jan/24 05:24
• Resolved: 18/May/22 16:20

Description

AbstractResourceUsage#getNodePartitionsSet and QueueCapacities#getNodePartitionsSet provide keySet, a mutable view on the HashMap's keys, that is subject to change. Iterating through an iterator that is modified by an other thread at the same time results in a ConcurrentModificationException as the following stacktrace shows:

2022-04-28 13:21:53,692 FATAL org.apache.hadoop.yarn.event.EventDispatcher: Error in handling event type NODE_LABELS_UPDATE to the Event Dispatcher
java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445)
    at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
    at com.google.common.collect.Sets$1$1.computeNext(Sets.java:758)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateQueueStatistics(CSQueueUtils.java:236)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.updateClusterResource(ParentQueue.java:1281)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updateNodeLabelsAndQueueResource(CapacityScheduler.java:2115)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1900)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:169)
    at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
    at java.lang.Thread.run(Thread.java:748)
 

Issue Links

• links to: GitHub Pull Request #4301