[HADOOP-2141] speculative execution start up condition based on completion time

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0
• Hadoop Flags: Reviewed
• Assignee: Andy Konwinski
• Reporter: Koji Noguchi
• Created: 01/Nov/07 20:03
• Updated: 19/Oct/12 13:57
• Resolved: 16/Jun/09 04:14

Description

We had one job with speculative execution hang.

4 reduce tasks were stuck with 95% completion because of a bad disk.

Devaraj pointed out
bq . One of the conditions that must be met for launching a speculative instance of a task is that it must be at least 20% behind the average progress, and this is not true here.

It would be nice if speculative execution also starts up when tasks stop making progress.

Devaraj suggested

Maybe, we should introduce a condition for average completion time for tasks in the speculative execution check.

Issue Links

• is related to: HADOOP-2211 The default task timeout for streaming should be large, but finite
• relates to: MAPREDUCE-92 Reduce task stuck at 95.71% for a long time and the speculative execution does not kick in
• relates to: MAPREDUCE-94 Speculative execution does not work properly
• relates to: MAPREDUCE-2039 Improve speculative execution