[FLINK-33053] Watcher leak in Zookeeper HA mode

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.17.0, 1.18.0, 1.17.1
• Fix Version/s: 1.19.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yangze Guo
• Reporter: Yangze Guo
• Created: 07/Sep/23 03:45
• Updated: 23/Nov/23 08:45
• Resolved: 19/Sep/23 05:37

Description

We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA mode. TM's watches on the leader of JobMaster has not been stopped after job finished.

Here is how we re-produce this issue:

• Start a session cluster and enable Zookeeper HA mode.
• Continuously and concurrently submit short queries, e.g. WordCount to the cluster.
• echo -n wchp | nc {zk host} {zk port} to get current watches.

We can see a lot of watches on /flink/{cluster_name}/leader/{job_id}/connection_info.

Issue Links

• relates to: ZOOKEEPER-4625 No reliable way to remove watcher without interfering others in same watch mode and on same path
• links to: GitHub Pull Request #23415
• links to: GitHub Pull Request #23773