[MAPREDUCE-3895] Speculative execution algorithm in 1.0 is too pessimistic in many cases

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Component/s: jobtracker, performance
• Assignee: Unassigned
• Reporter: Nathan Roberts
• Created: 22/Feb/12 16:09
• Updated: 22/Feb/12 20:05

Description

We are seeing many instances where largish jobs are ending up with 30-50% of reduce tasks being speculatively re-executed. This can be a significant drain on cluster resources.

The primary reason is due to the way progress in the reduce phase can make huge jumps in a very short amount of time. This fact leads the speculative execution code to think lots of tasks have fallen way behind the average when in fact they haven't

The important piece of the algorithm is essentially:

• Am I more than 20% behind the average progress?
• Have I been running for at least a minute?
• Have any tasks completed yet?

Unfortunately, a set of reduce tasks which spend a couple of minutes in the Copy phase, and very little time in the Sort phase, will trigger all these conditions for a large percentage of the reduce tasks. (the tasks' progress jump from 33% to 66% almost instantly which then triggers the speculation). I've seen this on several very large jobs which spend about 2 minutes in Copy, a few seconds in Sort, and 40 minutes in Reduce. These jobs launch about 30-40% additional reduce tasks which then run for almost the full 40 minutes.

This area becomes more plugable in MRv2 but for 1.0 it would be good if some portion of this algorithm could be configurable so that a job could have some degree of control (just disabling speculative execution is not really an option).