[YARN-3054] Preempt policy in FairScheduler may cause mapreduce job never finish

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.6.0
• Component/s: fairscheduler
• Assignee: Unassigned
• Reporter: Peng Zhang
• Created: 13/Jan/15 08:00
• Updated: 14/Feb/17 21:39
• Resolved: 31/Aug/16 03:45

Description

Preemption policy is related with schedule policy now. Using comparator of schedule policy to find preemption candidate cannot guarantee a subset of containers never be preempted. And this may cause tasks to be preempted periodically before they finish. So job cannot make any progress.

I think preemption in YARN should got below assurance:
1. Mapreduce jobs can get additional resources when others are idle;
2. Mapreduce jobs for one user in one queue can still progress with its min share when others preempt resources back.

Maybe always preempt the latest app and container can get this?

Issue Links

• is part of: YARN-4752 FairScheduler should preempt for a ResourceRequest and all preempted containers should be on the same node
• is related to: MAPREDUCE-6849 Preemption by container priority can be problematic for MapReduce