[YARN-2989] Better Load Balancing in Fair Scheduler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.5.0
• Component/s: fairscheduler
• Assignee: Unassigned
• Reporter: Kannan Rajah
• Created: 23/Dec/14 21:20
• Updated: 24/Dec/14 02:20
• Resolved: 24/Dec/14 02:20

Description

While porting Fair Scheduler from MR1, we seem to have changed the logic behind task distribution across nodes (MAPREDUCE-3451).

In MR1, a load factor was computed using runnableMaps/totalMapSlots and this was used to determine how many tasks need to be given to a node such that the overall cluster load is evenly distributed. In one heartbeat, we could assign multiple tasks. In YARN, we have the option to assign multiple tasks to a node, but this is disabled by default (YARN-302). Even when it is enabled, the number of tasks to assign is statically configured. So it won't ensure that load is evenly distributed. Why not bring back the load factor based check? Any reason why it was not done? This is actually more relevant with label based scheduling.

If there are no objections, I would like to implement it for both normal and label based scheduling scenarios.