[YARN-5342] Improve non-exclusive node partition resource allocation in Capacity Scheduler

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Hadoop Flags: Reviewed
• Assignee: Sunil G
• Reporter: Wangda Tan
• Created: 08/Jul/16 21:33
• Updated: 05/Apr/19 18:18
• Resolved: 03/Aug/16 18:32

Description

In the previous implementation, one non-exclusive container allocation is possible when the missed-opportunity >= #cluster-nodes. And missed-opportunity will be reset when container allocated to any node.

This will slow down the frequency of container allocation on non-exclusive node partition: When a non-exclusive partition=x has idle resource, we can only allocate one container for this app in every X=nodemanagers.heartbeat-interval secs for the whole cluster.

In this JIRA, I propose a fix to reset missed-opporunity only if we have >0 pending resource for the non-exclusive partition OR we get allocation from the default partition.

Issue Links

• causes: YARN-9449 Non-exclusive labels can create reservation loop on cluster without unlabeled node