[HADOOP-4943] fair share scheduler does not utilize all slots if the task trackers are configured heterogeneously

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.1
• Hadoop Flags: Reviewed

Release Note

HADOOP-4943: Fixed fair share scheduler to utilize all slots when the task trackers are configured heterogeneously.

• Assignee: Zheng Shao
• Reporter: Zheng Shao
• Created: 24/Dec/08 22:19
• Updated: 08/Jul/09 16:41
• Resolved: 13/Jan/09 06:07

Description

There is some code in the fairshare scheduler that tries to make the load across the whole cluster the same.
That piece of code will break if the task trackers are configured differently. Basically, we will stop assigning more tasks to tasks trackers that have tasks above the cluster average, but we may still want to do that because other task trackers may have less slots.

We should change the code to maintain a cluster-wide slot usage percentage (instead of absolute number of slot usage) to make sure the load is evenly distributed.