[YARN-1680] availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0, 2.3.0
• Component/s: capacityscheduler

Environment

SuSE 11 SP2 + Hadoop-2.3

• Assignee: Unassigned
• Reporter: Rohith Sharma K S
• Created: 24/Jan/14 08:21
• Updated: 07/Jan/17 01:52

Description

There are 4 NodeManagers with 8GB each.Total cluster capacity is 32GB.Cluster slow start is set to 1.

Job is running reducer task occupied 29GB of cluster.One NodeManager(NM-4) is become unstable(3 Map got killed), MRAppMaster blacklisted unstable NodeManager(NM-4). All reducer task are running in cluster now.

MRAppMaster does not preempt the reducers because for Reducer preemption calculation, headRoom is considering blacklisted nodes memory. This makes jobs to hang forever(ResourceManager does not assing any new containers on blacklisted nodes but returns availableResouce considers cluster free memory).

Issue Links

• Is contained by: YARN-2848 (FICA) Applications should maintain an application specific 'cluster' resource to calculate headroom and userlimit
• is duplicated by: MAPREDUCE-5928 Deadlock allocating containers for mappers and reducers
• is related to: MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom