[YARN-4676] Automatic and Asynchronous Decommissioning Nodes Status Tracking

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: resourcemanager
• Labels: features
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Zhi
• Reporter: Daniel Zhi
• Created: 05/Feb/16 21:55
• Updated: 25/Oct/19 20:27
• Resolved: 18/Aug/16 14:43

Description

YARN-4676 implements an automatic, asynchronous and flexible mechanism to graceful decommission
YARN nodes. After user issues the refreshNodes request, ResourceManager automatically evaluates
status of all affected nodes to kicks out decommission or recommission actions. RM asynchronously
tracks container and application status related to DECOMMISSIONING nodes to decommission the
nodes immediately after there are ready to be decommissioned. Decommissioning timeout at individual
nodes granularity is supported and could be dynamically updated. The mechanism naturally supports multiple
independent graceful decommissioning “sessions” where each one involves different sets of nodes with
different timeout settings. Such support is ideal and necessary for graceful decommission request issued
by external cluster management software instead of human.

DecommissioningNodeWatcher inside ResourceTrackingService tracks DECOMMISSIONING nodes status automatically and asynchronously after client/admin made the graceful decommission request. It tracks DECOMMISSIONING nodes status to decide when, after all running containers on the node have completed, will be transitioned into DECOMMISSIONED state. NodesListManager detect and handle include and exclude list changes to kick out decommission or recommission as necessary.

Issue Links

• relates to: YARN-5464 Server-Side NM Graceful Decommissioning with RM HA