[YARN-10352] Skip schedule on not heartbeated nodes in Multi Node Placement

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.4.0
• Fix Version/s: 3.4.0
• Component/s: scheduler
• Labels: capacityscheduler multi-node-placement
• Target Version/s: 3.3.0, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Prabhu Joseph
• Reporter: Prabhu Joseph
• Created: 16/Jul/20 15:39
• Updated: 12/Feb/24 01:14
• Resolved: 03/Feb/21 21:56

Description

When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM Active Nodes will be still having those stopped nodes until NM Liveliness Monitor Expires after configured timeout (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, Multi Node Placement assigns the containers on those nodes. They need to exclude the nodes which has not heartbeated for configured heartbeat interval (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to Asynchronous Capacity Scheduler Threads. (CapacityScheduler#shouldSkipNodeSchedule)

Repro:

1. Enable Multi Node Placement (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery Enabled (yarn.node.recovery.enabled)

2. Have only one NM running say worker0

3. Stop worker0 and start any other NM say worker1

4. Submit a sleep job. The containers will timeout as assigned to stopped NM worker0.

Issue Links

• is duplicated by: YARN-8557 Exclude lagged/unhealthy/decommissioned nodes in async allocating thread
• is related to: YARN-10357 Proactively relocate allocated containers from a stopped node
• relates to: YARN-10572 Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.