[YARN-80] Support delay scheduling for node locality in MR2's capacity scheduler

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.2-alpha, 0.23.6
• Component/s: capacityscheduler
• Assignee: Arun Murthy
• Reporter: Todd Lipcon
• Created: 19/Oct/11 07:15
• Updated: 07/Nov/13 13:29
• Resolved: 07/Sep/12 05:12

Description

The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.

Issue Links

• is related to: MAPREDUCE-4305 Implement delay scheduling in capacity scheduler for improving data locality
• supercedes: YARN-284 YARN capacity scheduler doesn't spread MR tasks evenly on an underutilized cluster