Assuming
HADOOP-2119 provides better data structures for handling running TIPs, finding new speculative tasks can be further optimized. Two of which could be
1)
conf.getMapSpeculativeExecution() and
conf.getReduceSpeculativeExecution() should be moved to
JobInProgress. A simple check for this boolean can prove useful before checking for speculative tasks. This will be useful for jobs with large maps and reducers where scanning all the TIPs can be costly.
2) Since the progress of a TIP changes only when
TaskInProgress.recomputeProgress() is invoked, it makes more sense to check for speculation in
JobInProgress.updateTaskStatus() and move the TIPs that can be speculated to the front of the running queue.
Improvement
Open
Major
Optimize finding of speculative tasks
