[MAPREDUCE-543] large pending jobs hog resources

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 0.21.0
• Assignee: Matei Alexandru Zaharia
• Reporter: Joydeep Sen Sarma
• Created: 08/Dec/08 17:14
• Updated: 24/Aug/10 21:13
• Resolved: 10/Nov/09 08:02

Description

observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:

• large job comes along and needs to wait for a while for other large jobs.
• slots are slowly transfered from one large job to another
• small tasks keep waiting forever.

is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs