[MAPREDUCE-339] JobTracker should give preference to failed tasks over virgin tasks so as to terminate the job ASAP if it is eventually going to fail.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Devaraj Das
• Reporter: Gautam Kowshik
• Created: 01/Jun/09 12:52
• Updated: 21/Jul/10 06:49

Description

Case in point... I have 1585 maps and 160 slots (40 nodes). The job is such that all maps fail within 2-3 minutes. The job takes forever to realise that the job is bad. It took 2526 failures for it to reach 4 failed attempts for a task.

As I understand, currently the JT prefers a failed task if and only if a task tracker with a split replica for that map came asking for a task. In fact there may not be a single TT at all in the mapred cluster which has a replica for the splits used in this job (pre-0.20). This delays the job failure by a lot and hence degrades cluster utilization as a whole. If i'm on a shared cluster with many jobs waiting on it to fail, it's bad.

The JT should prefer a failed task a lot earlier than waiting for a data local TT to come around asking.