diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md new file mode 100644 index 0000000..c357b2b --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/GracefulDecommission.md @@ -0,0 +1,114 @@ + + +Graceful Decommission of Yarn Nodes +=============== + +* [Overview](#overview) +* [Features](#features) + * [NodesListManager detects and handles include and exclude list changes](#nodeslistmanager-detects-and-handles-include-and-exclude-list-changes) + * [RMNode handles decommission events](#rmnode-handles-decommission-events) + * [Automatic and asynchronous tracking of decommissioning nodes status](#automatic-and-asynchronous-tracking-of-decommissioning-nodes-status) + * [Per-Node decommission timeout support](#per-node-decommission-timeout-support) +* [Configuration](#configuration) + +Overview +-------- + +Graceful Decommission of Yarn Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED. + +Features +-------- + +###NodesListManager detects and handles include and exclude list changes + +"yarn rmadmin -refreshNodes -g [timeout in seconds]" notifies NodesListManager to detect and handle include and exclude hosts changes. NodesListManager loads excluded hosts from the exclude file as specified through the "yarn.resourcemanager.nodes.exclude-path" configuration in yarn-site.xml. + +NodesListManager inspects and compares status of RMNodes in resource manager and the exclude list, and apply necessary actions based on following rules: + +* Recommission DECOMMISSIONED or DECOMMISSIONING nodes that are no longer excluded; +* Gracefully decommission excluded nodes that are not already in DECOMMISSIONED nor +DECOMMISSIONING state; +* Forcefully decommission excluded nodes that are not already in DECOMMISSIONED state should -g flag is not specified. + +Accordingly, RECOMMISSION, GRACEFUL_DECOMMISSION or DECOMMISSION RMNodeEvent will be sent to the RMNode. + +###Per-Node decommission timeout support + +To support flexible graceful decommission of nodes using different timeout through +single or multiple refreshNodes requests, HostsFileReader supports optional timeout value +after each hostname (or ip) in the exclude host file. The effective decommissioning timeout +to use for a particular host is based on following priorities: +* Use the timeout for the particular host if specified in exclude host file; +* Use the timeout in "yarn rmadmin -refreshNodes -g [timeout in seconds]" if specified; +* Use the default timeout specified through "yarn.resourcemanager.decommissioning.default.timeout" configuration. + +NodesListManager decides the effective timeout to use and set it on individual RMNode. +The timeout could also be dynamically adjusted through "yarn rmadmin -refreshNodes -g [timeout in seconds]" command. NodesListManager will resolve the effective timeout to use and update RMNode as necessary of the new timeout. Change of timeout does not reset the ongoing decommissioning but only affect the evaluation of whether the node has reached decommissioning timeout. + +Here is a sample yarn.nodes.exclude.xml. + +```xml + + + host1 + host2123 + host3-1 + host4, host5,host61800 + +``` + +###RMNode handles decommission events + +Upon receiving GRACEFUL_DECOMMISSION event, the RMNode will save the decommissioning timeout if specified, update metrics for graceful decommission and preserve its original total capacity, and transition into DECOMMISSIONING state. + +Resources will be dynamically and periodically updated on DECOMMISSIONING RMNode so that scheduler won't be scheduling new containers on them due to no available resources. + +###Automatic and asynchronous tracking of decommissioning nodes status +**DecommissioningNodeWatcher** is the YARN component that tracks DECOMMISSIONING nodes +status automatically and asynchronously after client/admin made the graceful decommission +request. NM periodically send RM heart beat with it latest container status. +DecommissioningNodeWatcher tracks heartbeat updates on all DECOMMISSIONING nodes to decide when, +after all running containers on the node have completed, will be transitioned into DECOMMISSIONED state +after which NodeManager will be told to shutdown. + +Under MR application, a node, after completes all its containers, may still serve it map output data +during the duration of the application for reducers. The YARN graceful decommission +mechanism keeps such DECOMMISSIONING nodes until all involved applications complete. +It could be however undesirable under long-running applications scenario where a bunch of +"idle" nodes might stay around for long period of time. DecommissioningNodeWatcher +balances such concern with a timeout --- a DECOMMISSIONING node will be DECOMMISSIONED +no later than decommissioning timeout regardless of running containers or applications. +If running containers finished earlier, it continues waiting for applications to finish +until the decommissioning timeout. When decommissioning timeout reaches, the node +will be decommissioned regardless. The node will be deactivated and owning tasks will +be rescheduled as necessary. + +Status of all decommissioning node are logged periodically (every 20 seconds) in resource manager logs. +Following are the sub-status of a decommissioning node: + +* NONE --- Node is not in DECOMMISSIONING state. +* WAIT_CONTAINER --- wait for running containers to complete. +* WAIT_APP --- wait for running application to complete (after all containers complete) +* TIMEOUT --- Timeout waiting for either containers or applications to complete +* READY --- Nothing to wait, ready to be decommissioned +* DECOMMISSIONED --- The node has already been decommissioned + +Configuration +-------- + +Property | Value +----- | ------ +yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs | Timeout in seconds for YARN node graceful decommission. This is the maximal time to wait for running containers and applications to complete before transition a DECOMMISSIONING node into DECOMMISSIONED. The default value is 3600 seconds. Negative value (like -1) is handled as infinite timeout. +yarn.resourcemanager.decommissioning-nodes-watcher.poll-interval-secs | Period in seconds of the poll timer task inside DecommissioningNodesWatcher to identify and take care of DECOMMISSIONING nodes missing regular heart beat. The default value is 20 seconds.