[YARN-8061] An application may preempt itself in case of minshare preemption

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.0, 2.8.3, 3.0.0
• Component/s: fairscheduler
• Assignee: Zhaohui Xin
• Reporter: Yufei Gu
• Created: 22/Mar/18 00:24
• Updated: 13/Feb/19 07:22

Description

Assume a leaf queue A's minshare is 10G memory and fairshare is 12G. It used 4G, so its minshare-staved resources is 6G and will be distributed to all its apps. Assume there are 4 apps a1, a2, a3, a4 inside, who demand 3G, 2G, 1G, and 0.5G. a1 gets 3G minshare-starved resources, a2 gets 2G, a3 get 1G, they are all considered as starved apps except a4 who doesn't get any.

An app can preempt another under the same queue due to minshare starvation. For example, a1 can preempt a4 if a4 uses more resources than its fair share, which is 3G(12G/4). If a1 itself used more than 3G memory, it will preempt itself! I will create a unit test later.

The solution would check application's fair share while distributing minshare starvation, more details in method FSLeafQueue#updateStarvedAppsMinshare().

Issue Links

• is related to: YARN-9277 Add more restrictions In FairScheduler Preemption
• is related to: YARN-9066 Deprecate Fair Scheduler min share