[YUNIKORN-2908] Remove associated metrics when queue is removed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Labels: pull-request-available
• Target Version: 1.7.0
• Assignee: Hengzhe Guo
• Reporter: Hengzhe Guo
• Created: 08/Oct/24 21:49
• Updated: 29/Oct/24 19:56
• Resolved: 29/Oct/24 19:56

Description

1. after a queue is removed, its metrics will continue to be reported by prometheus. This is fine with metrics like allocated resource because they will just be 0, but it won't make sense for guaranteed and max resources, giving wrong impression that there are still resource given to the queue. I propose to unregister all this queue's metrics when it's removed.

2. If queue is not removed but guaranteed or max resource config is removed, or just a resource type is removed from the config, the metrics are also not cleaned up. these metrics are only updated when there's a new valid value, but not 'null' value. I propose to always delete all existing guaranteed and max resources metrics of the queue then add back the new values, every time we apply the configs.

Issue Links

• links to: GitHub Pull Request #987