[MAPREDUCE-4305] Implement delay scheduling in capacity scheduler for improving data locality

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Mayank Bansal
• Reporter: Mayank Bansal
• Created: 01/Jun/12 22:14
• Updated: 08/Oct/13 03:00

Description

Capacity Scheduler data local tasks are about 40%-50% which is not good.
While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster.

I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality.
http://radlab.cs.berkeley.edu/publication/308

After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster.

Thanks,
Mayank

Issue Links

• relates to: YARN-80 Support delay scheduling for node locality in MR2's capacity scheduler