[YARN-6956] preemption may only consider resource requests for one node

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.9.0, 3.0.0-beta1
• Component/s: fairscheduler

Environment

CDH 5.11.0

• Assignee: Steven Rand
• Reporter: Steven Rand
• Created: 06/Aug/17 03:36
• Updated: 08/Feb/18 23:34

Description

I'm observing the following series of events on a CDH 5.11.0 cluster, which seem to be possible after YARN-6163:

1. An application is considered to be starved, so FSPreemptionThread calls identifyContainersToPreempt, and that calls FSAppAttempt#getStarvedResourceRequests to get a list of ResourceRequest instances that are enough to address the app's starvation.

2. The first ResourceRequest that getStarvedResourceRequests sees is enough to address the app's starvation, so we break out of the loop over appSchedulingInfo.getAllResourceRequests() after only one iteration: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L1180. We return only this one ResourceRequest back to the identifyContainersToPreempt method.

3. It turns out that this particular ResourceRequest happens to have a value for getResourceName that identifies a specific node in the cluster. This causes preemption to only consider containers on that node, and not the rest of the cluster.

kasha, does that make sense? I'm happy to submit a patch if I'm understanding the problem correctly.

Issue Links

• is related to: YARN-7903 Method getStarvedResourceRequests() only consider the first encountered resource
• relates to: YARN-7655 Avoid AM preemption caused by RRs for specific nodes or racks