[YARN-56] Handle container requests that request more resources than currently available in the cluster

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.2-alpha, 0.23.3
• Component/s: resourcemanager
• Assignee: Unassigned
• Reporter: Hitesh Shah
• Created: 22/Aug/12 23:35
• Updated: 25/Jul/17 15:27

Description

In heterogenous clusters, a simple check at the scheduler to check if the allocation request is within the max allocatable range is not enough.

If there are large nodes in the cluster which are not available, there may be situations where some allocation requests will never be fulfilled. Need an approach to decide when to invalidate such requests. For application submissions, there will need to be a feedback loop for applications that could not be launched. For running AMs, AllocationResponse may need to augmented with information for invalidated/cancelled container requests.

Issue Links

• duplicates: MAPREDUCE-3946 If a resource requirement is higher than available on any node, job should fail early
• duplicates: YARN-2555 Effective max-allocation-* should consider biggest node
• is duplicated by: YARN-2604 Scheduler should consider max-allocation-* in conjunction with the largest node
• relates to: YARN-394 RM should be able to return requests that it cannot fulfill
• relates to: YARN-389 Infinitely assigning containers when the required resource exceeds the cluster's absolute capacity
• relates to: YARN-6867 AbstractYarnScheduler reports the configured maximum resources, instead of the actual, even after the configured waittime