[YARN-6901] A CapacityScheduler app->LeafQueue deadlock found in branch-2.8

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Target Version/s: 2.8.3
• Assignee: Wangda Tan
• Reporter: Wangda Tan
• Created: 28/Jul/17 22:36
• Updated: 14/Oct/19 18:45

Description

Stacktrace:

Thread 22068: (state = BLOCKED)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.getParent() @bci=0, line=185 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getQueuePath() @bci=8, line=262 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator.getCSAssignmentFromAllocateResult(org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.ContainerAllocation, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer) @bci=183, line=80 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator.assignContainers(org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.SchedulingMode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer) @bci=204, line=747 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.ContainerAllocator.assignContainers(org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.SchedulingMode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer) @bci=16, line=49 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.assignContainers(org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.SchedulingMode, org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainer) @bci=61, line=468 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode, org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.SchedulingMode) @bci=148, line=876 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode) @bci=157, line=1149 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEvent) @bci=266, line=1277 (Compiled frame)

================

 Thread 22124: (state = BLOCKED)
 - org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.getReservedContainers() @bci=0, line=336 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.FifoCandidatesSelector.preemptFrom(org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp, org.apache.hadoop.yarn.api.records.Resource, java.util.Map, java.util.List, org.apache.hadoop.yarn.api.records.Resource, java.util.Map, org.apache.hadoop.yarn.api.records.Resource) @bci=61, line=277 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.FifoCandidatesSelector.selectCandidates(java.util.Map, org.apache.hadoop.yarn.api.records.Resource, org.apache.hadoop.yarn.api.records.Resource) @bci=374, line=138 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.containerBasedPreemptOrKill(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue, org.apache.hadoop.yarn.api.records.Resource) @bci=264, line=342 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.editSchedule() @bci=34, line=202 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.invokePolicy() @bci=4, line=81 (Compiled frame)
 - org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor$PreemptionChecker.run() @bci=23, line=92 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Issue Links

• is related to: YARN-6917 Queue path is recomputed from scratch on every allocation