[YARN-9278] Shuffle nodes when selecting to be preempted nodes

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: fairscheduler
• Assignee: Zhaohui Xin
• Reporter: Zhaohui Xin
• Created: 03/Feb/19 13:10
• Updated: 26/Feb/20 06:14

Description

We should shuffle the nodes to avoid some nodes being preempted frequently. 

Also, we should limit the num of nodes to make preemption more efficient.

Just like this,

// we should not iterate all nodes, that will be very slow
long maxTryNodeNum = context.getPreemptionConfig().getToBePreemptedNodeMaxNumOnce();

if (potentialNodes.size() > maxTryNodeNum){
  Collections.shuffle(potentialNodes);
  List<FSSchedulerNode> newPotentialNodes = new ArrayList<FSSchedulerNode>();

for (int i = 0; i < maxTryNodeNum; i++){
  newPotentialNodes.add(potentialNodes.get(i));
}
potentialNodes = newPotentialNodes;