[HADOOP-5884] Capacity scheduler should account high memory jobs as using more capacity of the queue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.1
• Hadoop Flags: Reviewed

Release Note

Fixes Capacity scheduler to account more capacity of a queue for a high memory job. Done by considering these jobs to
take more slots proportionally with respect to a slot's default memory size.

• Assignee: Vinod Kumar Vavilapalli
• Reporter: Hemanth Yamijala
• Created: 21/May/09 16:30
• Updated: 08/Jul/09 16:40
• Resolved: 05/Jun/09 06:01

Description

Currently, when a high memory job is scheduled by the capacity scheduler, each task scheduled counts only once in the capacity of the queue, though it may actually be preventing other jobs from using spare slots on that node because of its higher memory requirements. In order to be fair, the capacity scheduler should proportionally (with respect to default memory) account high memory jobs as using a larger capacity of the queue.

Issue Links

• blocks: MAPREDUCE-517 The capacity-scheduler should assign multiple tasks per heartbeat
• blocks: MAPREDUCE-516 Fix the 'cluster drain' problem in the Capacity Scheduler wrt High RAM Jobs
• incorporates: HADOOP-5934 testHighRamJobWithSpeculativeExecution needs some changes
• is blocked by: HADOOP-5932 MemoryMatcher logs 0 as freeMemOnTT even though there are free slots available on TaskTraker