[YUNIKORN-2203] Possible log spew in UGM code

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 1.5.0
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 27/Nov/23 22:50
• Updated: 20/Mar/24 14:32
• Resolved: 28/Nov/23 11:34

Description

QueueTracker.canRunApp() can flood the logs in various ways.

1) This is always printed on DEBUG level:

	log.Log(log.SchedUGM).Debug("Checking can run app",
		zap.Int("tracking type", int(trackType)),
		zap.String("queue path", qt.queuePath),
		zap.String("application", applicationID),
		zap.Strings("hierarchy", hierarchy))

This is called in every cycle as long as the application is in Accepted state and can truly cause problems on DEBUG level. It does not add too much value, so I suggest removing it.

2) "maxapplications" is hit:

		log.Log(log.SchedUGM).Warn("can't run app as allowing new application to run would exceed configured max applications limit of specific user/group",
			zap.Int("tracking type", int(trackType)),
			zap.String("queue path", qt.queuePath),
			zap.Int("current running applications", len(qt.runningApplications)),
			zap.Uint64("max running applications", qt.maxRunningApps))

This can be useful, but we can't afford logging this constantly. Possible approaches:
1) Remove it anyway
2) Rate limit
3) Log once per applicationID, then log it again when finally the application is allowed to run

Issue Links

• links to: GitHub Pull Request #736