[MAPREDUCE-6513] MR job got hanged forever when one NM unstable for some time

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Component/s: applicationmaster, resourcemanager
• Target Version/s: 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Varun Saxena
• Reporter: Bob.zhao
• Due: 15/Oct/15
• Created: 15/Oct/15 15:32
• Updated: 30/Aug/16 01:15
• Resolved: 13/May/16 21:56

Description

when job is in-progress which is having more tasks,one node became unstable due to some OS issue.After the node became unstable, the map on this node status changed to KILLED state.

Currently maps which were running on unstable node are rescheduled, and all are in scheduled state and wait for RM assign container.Seen ask requests for map till Node is good (all those failed), there are no ask request after this. But AM keeps on preempting the reducers (it's recycling).

Finally reducers are waiting for complete mappers and mappers did n't get container..

My Question Is:
============
why map requests did not sent AM ,once after node recovery.?

Issue Links

• is related to: MAPREDUCE-6541 Exclude scheduled reducer memory when calculating available mapper slots from headroom to avoid deadlock
• relates to: MAPREDUCE-5507 MapReduce reducer ramp down is suboptimal with potential job-hanging issues
• relates to: MAPREDUCE-6514 Job hangs as ask is not updated after ramping down of all reducers
• relates to: MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom