[YARN-11421] Graceful Decommission ignores launched containers and gets deactivated before timeout

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1, 3.3.1, 3.3.4
• Component/s: resourcemanager
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Abhishek Dixit
• Created: 19/Jan/23 10:52
• Updated: 28/Aug/23 13:56

Description

During Graceful Decommission, a Node gets deactivated before timeout even though there are launched containers on that node.

We have observed cases when graceful decommission signal is sent to node and Containers are launched at NodeManager and at the same time, in such cases ResourceManager moves the node from Decommissioning to Decommissioned state because launced containers are not checked in DecommissioningNodesWatcher.

We will suggest waiting for yarn.resourcemanager.decommissioning-nodes-watcher.delay-ms to complete before marking node ready to be decommissioned. No delay if set to 0. Expire interval should not be configured more than RM_AM_EXPIRY_INTERVAL_MS.

Issue Links

• links to: GitHub Pull Request #5905