[MAPREDUCE-718] Support for per-phase speculative execution

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.21.0
• Component/s: jobtracker
• Assignee: Unassigned
• Reporter: Devaraj Das
• Created: 07/Jul/09 04:21
• Updated: 12/Dec/10 17:10

Description

It would be good to have support for per-phase speculative execution where the algorithm looks at the current phase of a task, and compares with the other tasks in the same phase before deciding to launch a speculative task. That would have the following benefits:
1) Support for jobs where map tasks progresses jumps from 0% to 100%. This is true for some jobs like randomwriter. Today, we would launch speculative tasks for such jobs (assuming that the tasks are not making progress). But most of them would be unnecessary.
2) In reality, for reduces, the three phases are quite different from each other, and they take different times too. We should see better results when we look at per-phase speculation.

Issue Links

• is blocked by: MAPREDUCE-901 Move Framework Counters into a TaskMetric structure
• relates to: MAPREDUCE-2216 speculation should normalize progress rates based on amount of input data