[MAPREDUCE-262] Optimize finding of speculative tasks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Amar Kamat
• Created: 22/Feb/08 11:16
• Updated: 17/Jul/14 21:05
• Resolved: 17/Jul/14 21:05

Description

Assuming HADOOP-2119 provides better data structures for handling running TIPs, finding new speculative tasks can be further optimized. Two of which could be
1) conf.getMapSpeculativeExecution() and conf.getReduceSpeculativeExecution() should be moved to JobInProgress. A simple check for this boolean can prove useful before checking for speculative tasks. This will be useful for jobs with large maps and reducers where scanning all the TIPs can be costly.
2) Since the progress of a TIP changes only when TaskInProgress.recomputeProgress() is invoked, it makes more sense to check for speculation in JobInProgress.updateTaskStatus() and move the TIPs that can be speculated to the front of the running queue.

Issue Links

• is blocked by: HADOOP-2119 JobTracker becomes non-responsive if the task trackers finish task too fast