Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.12.2
-
None
-
None
Description
The JobTracker should keep a track (with feedback from Reducers) of how many times a fetch for a particular map output failed. If this exceeds a certain threshold, then that map should be declared as lost, and should be reexecuted elsewhere. Based on the number of such complaints from Reducers, the JobTracker can blacklist the TaskTracker. This will make the framework reliable - it will take care of (faulty) TaskTrackers that sometimes always fail to serve up map outputs (for which exceptions are not properly raised/handled, for e.g., if the exception/problem happens in the Jetty server).
Attachments
Attachments
Issue Links
- duplicates
-
HADOOP-1454 Reducer doesn't track failed fetches and gets stuck
- Closed
- relates to
-
MAPREDUCE-79 Ignored IOExceptions from MapOutputLocation.java:getFile lead to hung reduces
- Open