[YARN-11202] Optimize ClientRMService.getApplications

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: yarn
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Tamas Domok
• Reporter: Tamas Domok
• Created: 28/Jun/22 08:19
• Updated: 28/Jan/24 05:16
• Resolved: 01/Jul/22 09:02

Description

YARN-11114 introduced an extra check in ClientRMService.getApplications() where it calculates a map (queue to a list of rmapps) inside a while loop which could have been calculated once before the while loop (which iterates over the apps).

Code walk-through:

queues:
 [q1, q2, q3]
apps:                 (apps running/non-running)
 [a1, a2, a3, a4, a5, a6, a7, a8, a9]

inside the while cycle:
  application:         (current iterator)
   [a5]
  foundApps:           (this is re-calculated every time, running apps)
    q1: [a1, a2]
    q2: [a4, a5, a6]
    q3: [a7, a9]
  runningAppsByQueues: (foundApps is already filtered by queues ... basically this concatenates the lists with extra steps)
    [a1, a2, a4, a5, a6, a7, a9]
  runningAppsById:
    [a5]

I think this could be simplified significantly, basically we just need a set of running app-ids (filtered by queues), which is calculated before the while loop, once / request. Then inside the while loop:

- if (!queues.contains(application.getQueue())) {
+ if (!runningApps.contains(application.getApplicationId()) && !queues.contains(application.getQueue())) {

The tests pass with this fix.

There is a potential bug inside the queryApplicationsByQueues, between the following time range: a new app is already in a queue, but not in the apps that was queried before:

apps = rmContext.getRMApps();
...
scheduler.getAppsInQueue(queue);

Issue Links

• links to: GitHub Pull Request #4508