[YARN-914] (Umbrella) Support graceful decommission of nodemanager

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.4-alpha
• Component/s: graceful
• Assignee: Junping Du
• Reporter: Luke Lu
• Created: 10/Jul/13 23:22
• Updated: 25/Oct/19 20:26

Description

When NMs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications.

Currently if a NM is decommissioned, all running containers on the NM need to be rescheduled on other NMs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well.

We propose to introduce a mechanism to optionally gracefully decommission a node manager.

Issue Links

• depends upon: YARN-311 Dynamic node resource configuration: core scheduler changes
• duplicates: YARN-671 Add an interface on the RM to move NMs into a maintenance state
• is blocked by: YARN-291 [Umbrella] Dynamic resource configuration
• is blocked by: HADOOP-9160 Adopt Jolokia as the JMX HTTP/JSON bridge.
• is related to: YARN-671 Add an interface on the RM to move NMs into a maintenance state
• is related to: YARN-4939 the decommissioning Node should keep alive if NM restart
• is related to: YARN-1996 Provide alternative policies for UNHEALTHY nodes.
• relates to: MAPREDUCE-5381 Support graceful decommission of tasktracker

Sub-Tasks

• RM to inform AMs when a container completed due to NM going offline -planned or unplanned: Resolved
• RMNode State Transition Update with DECOMMISSIONING state: Resolved
• Resource update during NM graceful decommission: Resolved
• Notify AM with containers (on decommissioning node) could be preempted after timeout.: Resolved
• New parameter or CLI for decommissioning node gracefully in RMAdmin CLI: Resolved
• Automatic and Asynchronous Decommissioning Nodes Status Tracking: Resolved
• UI changes for decommissioning node: Resolved
• RMNodeResourceUpdateEvent update from scheduler can lead to race condition: Resolved
• Document graceful decommission CLI and usage: Resolved
• Add -client|server argument for graceful decom: Resolved
• Server-Side NM Graceful Decommissioning with RM HA: Patch Available
• Server-Side NM Graceful Decommissioning subsequent call behavior: Open
• Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout: Open
• Client-side NM graceful decom is not triggered when jobs finish: Resolved
• Clarify DecommissionType.FORCEFUL comment: Resolved
• Document the current known issue with server-side NM graceful decom: Resolved
• Remove XML excludes file format: Resolved
• Better utilize gracefully decommissioning node managers: Open
• DecommissioningNodesWatcher should get lists of running applications on node from RMNode.: Resolved
• An easy method to exclude a nodemanager from the yarn cluster cleanly: Open
• Add option to graceful decommission to not wait for applications: Patch Available
• RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state: Resolved
• Avoid sending RMNodeResourceupdate event if resource is same: Resolved