Assuming HADOOP-2119 provides better data structures for handling running TIPs, finding new speculative tasks can be further optimized. Two of which could be
1) conf.getMapSpeculativeExecution() and conf.getReduceSpeculativeExecution() should be moved to JobInProgress. A simple check for this boolean can prove useful before checking for speculative tasks. This will be useful for jobs with large maps and reducers where scanning all the TIPs can be costly.
2) Since the progress of a TIP changes only when TaskInProgress.recomputeProgress() is invoked, it makes more sense to check for speculation in JobInProgress.updateTaskStatus() and move the TIPs that can be speculated to the front of the running queue.
Description
Assuming HADOOP-2119 provides better data structures for handling running TIPs, finding new speculative tasks can be further optimized. Two of which could be
1) conf.getMapSpeculativeExecution() and conf.getReduceSpeculativeExecution() should be moved to JobInProgress. A simple check for this boolean can prove useful before checking for speculative tasks. This will be useful for jobs with large maps and reducers where scanning all the TIPs can be costly.
2) Since the progress of a TIP changes only when TaskInProgress.recomputeProgress() is invoked, it makes more sense to check for speculation in JobInProgress.updateTaskStatus() and move the TIPs that can be speculated to the front of the running queue.
Improvement
Open
Major
Optimize finding of speculative tasks
