[FLINK-27420] Suspended SlotManager fails to re-register metrics when started again

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.5
• Fix Version/s: 1.14.5, 1.15.1, 1.16.0
• Component/s: Runtime / Coordination, Runtime / Metrics
• Labels: pull-request-available
• Assignee: Ben Augarten
• Reporter: Ben Augarten
• Created: 26/Apr/22 20:17
• Updated: 20/Jun/22 09:55
• Resolved: 20/Jun/22 09:55

Description

The symptom is that SlotManager metrics are missing (taskslotsavailable and taskslotstotal) when a SlotManager is suspended and then restarted. We noticed this issue when running 1.13.5, but I believe this impacts 1.14.x, 1.15.x, and master.

 

When a SlotManager is suspended, the metrics group is closed. When the SlotManager is started again, it makes an attempt to reregister metrics but that fails because the underlying metrics group is still closed 

 

I was able to trace through this issue by restarting zookeeper nodes in a staging environment and watching the JM with a debugger. 

 

A concise test, which currently fails, shows the expected behavior – https://github.com/apache/flink/compare/master...baugarten:baugarten/slot-manager-missing-metrics?expand=1

 

I am happy to provide a PR to fix this issue, but first would like to verify that this is not intended.

Issue Links

• links to: GitHub Pull Request #19607
• links to: GitHub Pull Request #19646