[YARN-4677] RMNodeResourceUpdateEvent update from scheduler can lead to race condition

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 3.2.0, 3.1.1, 2.9.2, 3.0.4
• Component/s: graceful, resourcemanager, scheduler
• Target Version/s: 3.2.0, 3.1.1, 2.9.2
• Hadoop Flags: Reviewed
• Assignee: Wilfred Spiegelenburg
• Reporter: Brook Zhou
• Created: 06/Feb/16 17:59
• Updated: 26/Feb/20 05:29
• Resolved: 04/Jun/18 23:49

Description

When a node is in decommissioning state, there is time window between completedContainer() and RMNodeResourceUpdateEvent get handled in scheduler.nodeUpdate (YARN-3223).

So if a scheduling effort happens within this window, the new container could still get allocated on this node. Even worse case is if scheduling effort happen after RMNodeResourceUpdateEvent sent out but before it is propagated to SchedulerNode - then the total resource is lower than used resource and available resource is a negative value.

Issue Links

• is blocked by: YARN-3223 Resource update during NM graceful decommission
• is duplicated by: YARN-5128 Investigate potential race condition in the scheduler nodeUpdate() method