[YUNIKORN-932] Improve logging reliability

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Labels: pull-request-available
• Target Version: 0.12.1
• Assignee: Kinga Marton
• Reporter: Kinga Marton
• Created: 03/Nov/21 12:07
• Updated: 21/Jan/22 21:48
• Resolved: 15/Nov/21 14:50

Description

There are cases wheen our logs are not relyable. For example we have seen in case of some escalations missing logs for hours and we don't know that it was because the cluster was not heavily used, or it was some issues.

We need to add some log messages for the following cases:

• log a message periodically if there are no pending requests
• log a message in case of recovery if we haven't heared anything from the API server for a while

Also it may worth to check the main log paths and extend the logs where it is necessary.

Issue Links

• causes: YUNIKORN-965 Fix race condition in k8s shim scheduler
• links to: GitHub Pull Request #326