[MAPREDUCE-6302] Preempt reducers after a configurable timeout irrespective of headroom

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
• Target Version/s: 2.8.0, 2.6.3, 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Karthik Kambatla
• Reporter: Shurong Mai
• Created: 30/Mar/15 04:26
• Updated: 12/Apr/20 03:14
• Resolved: 09/Oct/15 14:58

Description

I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue.
I think there is the similar issue for memory of a queue .

Issue Links

• breaks: MAPREDUCE-6514 Job hangs as ask is not updated after ramping down of all reducers
• breaks: MAPREDUCE-6689 MapReduce job can infinitely increase number of reducer resource requests
• is related to: MAPREDUCE-6513 MR job got hanged forever when one NM unstable for some time
• is related to: YARN-3485 FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies
• is related to: MAPREDUCE-5844 Add a configurable delay to reducer-preemption
• relates to: YARN-1680 availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.
• relates to: YARN-3446 FairScheduler headroom calculation should exclude nodes in the blacklist
• relates to: MAPREDUCE-6501 Improve reducer preemption