[HADOOP-248] locating map outputs via random probing is inefficient

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.1
• Fix Version/s: 0.12.0
• Assignee: Devaraj Das
• Reporter: Owen O'Malley
• Created: 24/May/06 06:25
• Updated: 02/May/13 02:29
• Resolved: 22/Feb/07 20:22

Description

Currently the ReduceTaskRunner polls the JobTracker for a random list of map tasks asking for their output locations. It would be better if the JobTracker kept an ordered log and the interface was changed to:

class MapLocationResults {
public int getTimestamp();
public MapOutputLocation[] getLocations();
}

interface InterTrackerProtocol {
...
MapLocationResults locateMapOutputs(int prevTimestamp);
}

with the intention that each time a ReduceTaskRunner calls locateMapOutputs, it passes back the "timestamp" that it got from the previous result. That way, reduces can easily find the new MapOutputs. This should help the "ramp up" when the maps first start finishing.

Issue Links

• depends upon: HADOOP-801 job tracker should keep a log of task completion and failure
• relates to: HADOOP-343 In case of dead task tracker, the copy mapouts try copying all mapoutputs from this tasktracker