[MAPREDUCE-5279] Jobs can deadlock if headroom is limited by cpu instead of memory

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.3-alpha
• Fix Version/s: 2.6.0
• Component/s: mrv2, scheduler
• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Peng Zhang
• Reporter: Peng Zhang
• Created: 28/May/13 08:13
• Updated: 01/Dec/14 03:09
• Resolved: 22/Sep/14 17:47

Description

YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't take into account virtual cores while scheduling reduce tasks.
This may cause more reduce tasks to be scheduled because memory is enough. And on a small cluster, this will end with deadlock, all running containers are reduce tasks but map phase is not finished.