[ZOOKEEPER-4428] ZooKeeper Server leaks "SyncThread" threads when leadership connection times out and is reestablished

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.6.3
• Component/s: server

Environment

1. On a follower node for an established ZooKeeper ensemble, issue the following command to determine number of SyncThreads:
   ps -T -p `pidof mdtzookeeper` | grep SyncThread | wc
2. Issue the following IP tables command on the leader to drop traffic coming from the follower used in Step 1:
   iptables -A INPUT -s <Follower IP Address> -j DROP
3. Watch the zookeeper logs on the nodes and wait for the connection to drop due to timeout.
4. Issue the following IP tables command on the leader to re-enable traffic coming from follower used in Step 1:
   iptables -D INPUT -s <Follower IP Address> -j DROP
5. Watch the zookeeper logs on the nodes and wait for the connection to the leader to reestablish.
6. On the follower node (used in Step 1), check the number of SyncThreads.  That value should have increased by one and stay pinned there indefinitely: 
   ps -T -p `pidof mdtzookeeper` | grep SyncThread | wc

• Assignee: Unassigned
• Reporter: Ryan Ruel
• Created: 15/Dec/21 16:20
• Updated: 15/Dec/21 16:23

Description

In a production environment with some connectivity problems it was found the ZooKeeper server was using over 1000 threads with name "SyncThread" (that were never being freed).

Looking through the server logs indicates that these nodes were experiencing connection timeouts to the leader.

A test environment (described below in the "environment" field of this ticket) showed that these connection timeouts are what seem to be leaking these threads.