[ZOOKEEPER-3997] Why Zookeeper 3.5.8 leader shutdown makes the follower not allow new sessions?

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Fix Version/s: 3.5.8
• Assignee: Unassigned
• Reporter: Vikramark
• Created: 07/Nov/20 23:45
• Updated: 08/Nov/20 02:07
• Resolved: 08/Nov/20 02:07

Description

There is a behavior difference between zookeeper 3.4.14 and 3.5.8.  In 3.5.8 given a cluster of 3 nodes, when the leader   node is shutdown, the follower does not seem to be in sync with the new leader. The clients are not able to establish session and requests gets queued up as outstanding requests.  We don't see this issue in case of 3.4.14 version.

Below are the details of how to recreate the issue:

 

With version 3.5.8:

 

Initial fresh setup of 3 node cluster:
 

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 3 Sent: 2 Connections: 1 Outstanding: 0 Zxid: 0x0 Mode: follower Node count: 5	Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 3 Sent: 2 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: leader Node count: 5 Proposal sizes last/min/max: -1/-1/-1	Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 2 Sent: 1 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: follower Node count: 5

 

After starting one session using zkCli.sh on Zoo1 node:

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 1/9/23 Received: 7 Sent: 6 Connections: 2 Outstanding: 0 Zxid: 0x100000001 Mode: follower Node count: 5	Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 4 Sent: 3 Connections: 1 Outstanding: 0 Zxid: 0x100000001 Mode: leader Node count: 5 Proposal sizes last/min/max: 36/36/36	Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 3 Sent: 2 Connections: 1 Outstanding: 0 Zxid: 0x100000001 Mode: follower Node count: 5

 

Note: We can see that Zxid is now consistent across all nodes.

 

After shutting down leader node zoo2. We can see ZOO3 became the Leader. For some reason the ZXID is not same between zoo1 and zoo3.

 

Start a new zkCli.sh session on same node (zoo1).  The session was not established, the cli client just keeps retrying and created many outstanding requests on zoo1. 

Expected behavior: The zoo1 should have same zxid as zoo3. The client session should be allowed to be created. No outstanding requests should be added. 

 

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/2 Received: 50 Sent: 43 Connections: 2 Outstanding: 6 Zxid: 0x100000001 Mode: follower Node count: 5	down	Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:07 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x200000000 Mode: leader Node count: 5 Proposal sizes last/min/max: -1/-1/-1

 

With version 3.4.14

First initial setup:

 

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x0 Mode: follower Node count: 4	Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: leader Node count: 4 Proposal sizes last/min/max: -1/-1/-1	Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: follower Node count: 4

 

After connecting with zkCli on ZOO1.

 

 

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/14/33 Received: 5 Sent: 4 Connections: 2 Outstanding: 0 Zxid: 0x100000001 Mode: follower Node count: 4	Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 2 Sent: 1 Connections: 1 Outstanding: 0 Zxid: 0x100000001 Mode: leader Node count: 4 Proposal sizes last/min/max: 36/36/36	Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 2 Sent: 1 Connections: 1 Outstanding: 0 Zxid: 0x100000001 Mode: follower Node count: 4

 

Note: The zkid is now same for all the nodes.

 

 

After shutting down leader node zoo2. We can see ZOO3 became the Leader. For some reason the ZXID is not same between zoo1 and zoo3 initially. ZOO3 has new zkid as a new epoch was created but zoo1 still had old zkid.

 

Now closed the existing zxcli and started a new zkCli.sh session on same node (zoo1).  This time session was established!.

 

Zoo1	Zoo2	Zoo3
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/1/4 Received: 8 Sent: 7 Connections: 2 Outstanding: 0 Zxid: 0x200000001 Mode: follower Node count: 4	down	Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT Latency min/avg/max: 0/0/0 Received: 3 Sent: 2 Connections: 1 Outstanding: 0 Zxid: 0x200000001 Mode: leader Node count: 4 Proposal sizes last/min/max: 36/36/36