[HADOOP-2220] Reduce tasks fail too easily because of repeated fetch failures

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.16.0
• Assignee: Amar Kamat
• Reporter: Christian Kunz
• Created: 17/Nov/07 02:45
• Updated: 08/Jul/09 16:52
• Resolved: 21/Dec/07 18:58

Description

Currently reduce tasks with more than MAX_FAILED_UNIQUE_FETCHES (= 5 hard-coded) failures to fetch output from different mappers will fail (I believe, introduced in HADOOP-1158)

This gives us some problems with longer running jobs with a large number of mappers in multiple waves:
Otherwise problem-less reduce tasks fail because of too many fetch failures due to resource contention, and new reduce tasks have to fetch all data from the already successfully executed mappers, introducing a lot of additional IO overhead. Also, the job will fail when the same reducer exhausts the maximum number of attempts.

The limit should be a function of number of mappers and/or waves of mappers, and should be more conservative (e.g. no need to let them fail when there are enough slots to start speculatively executed reducers and speculative execution is enabled). Also, we might consider to not count such a restart against the number of attempts.

Issue Links

• is part of: HADOOP-2247 Mappers fail easily due to repeated failures
• is related to: HADOOP-2247 Mappers fail easily due to repeated failures