[YARN-5543] ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0, 2.6.1
• Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
• Component/s: capacityscheduler, resourcemanager
• Labels: oct16-medium
• Target Version/s: 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Min Shen
• Reporter: Min Shen
• Created: 19/Aug/16 23:02
• Updated: 24/Apr/18 20:49
• Resolved: 11/May/17 20:50

Description

In SchedulingMonitor.java, when the service starts, it starts a checker thread to perform Capacity Scheduler's preemption. However, the implementation of this checker thread has the following issue:

while (!stopped && !Thread.currentThread().isInterrupted()) {
    ....
    try {
      Thread.sleep(monitorInterval)
    } catch (InterruptedException e) {
      ....
      break;
    }
}

The above code snippet will terminate the checker thread whenever it is interrupted.
We noticed in our cluster that this could lead to CapacityScheduler's preemption disabled unexpectedly due to the checker thread getting terminated.

We propose to use ScheduledExecutorService to improve the robustness of this part of the code to ensure the liveness of CapacityScheduler's preemption functionality.

Issue Links

• breaks: YARN-7084 TestSchedulingMonitor#testRMStarts fails sporadically