Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
Reviewed
Description
YARN-11114 introduced an extra check in ClientRMService.getApplications() where it calculates a map (queue to a list of rmapps) inside a while loop which could have been calculated once before the while loop (which iterates over the apps).
Code walk-through:
queues: [q1, q2, q3] apps: (apps running/non-running) [a1, a2, a3, a4, a5, a6, a7, a8, a9] inside the while cycle: application: (current iterator) [a5] foundApps: (this is re-calculated every time, running apps) q1: [a1, a2] q2: [a4, a5, a6] q3: [a7, a9] runningAppsByQueues: (foundApps is already filtered by queues ... basically this concatenates the lists with extra steps) [a1, a2, a4, a5, a6, a7, a9] runningAppsById: [a5]
I think this could be simplified significantly, basically we just need a set of running app-ids (filtered by queues), which is calculated before the while loop, once / request. Then inside the while loop:
- if (!queues.contains(application.getQueue())) { + if (!runningApps.contains(application.getApplicationId()) && !queues.contains(application.getQueue())) {
The tests pass with this fix.
There is a potential bug inside the queryApplicationsByQueues, between the following time range: a new app is already in a queue, but not in the apps that was queried before:
apps = rmContext.getRMApps();
...
scheduler.getAppsInQueue(queue);
Attachments
Issue Links
- links to