[YARN-3485] FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: fairscheduler
• Target Version/s: 2.7.1
• Hadoop Flags: Reviewed
• Assignee: Karthik Kambatla
• Reporter: Karthik Kambatla
• Created: 14/Apr/15 19:28
• Updated: 06/Jan/17 00:49
• Resolved: 29/Apr/15 04:10

Description

FairScheduler's headroom calculations consider the fairshare and cluster-available-resources, and the fairshare has maxResources. However, for Fifo and Fairshare policies, the fairshare is used only for memory and not cpu. So, the scheduler ends up showing a higher headroom than is actually available. This could lead to applications waiting for resources far longer than then intend to. e.g. MAPREDUCE-6302.

Issue Links

• is duplicated by: YARN-3891 Apps are starved for resource due to default FairSharePolicy in Fair Scheduler
• relates to: MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom