[ZOOKEEPER-3830] After add a new node, zookeeper cluster won't commit any proposal if this new node is leader

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.6.1, 3.5.8
• Fix Version/s: 3.5.9, 3.7.0, 3.6.2
• Component/s: server

Environment

Zookeeper 3.5.8

JDK 1.8

• Assignee: Mate Szalay-Beko
• Reporter: Keli Wang
• Created: 15/May/20 17:18
• Updated: 10/Sep/20 10:43
• Resolved: 03/Jun/20 08:03

Description

I have a zookeeper cluster with 3 nodes, node3 is the leader of the cluster.

 

server.1=node1
server.2=node2
server.3=node3 # current leader

With dynamic reconfiguration disabled, I scale this cluster to 4 nodes with 2 steps:

1. Start node4 with new config, now node4 is a follower.
2. Modify config and restart node1, node2 and node3 one by one.

The new cluster config is:

server.1=node1
server.2=node2
server.3=node3 
server.4=node4 # current leader

After restart, node4 is the leader of this cluster. But I cannot connect to this cluster using zkCli now, zkCli keeps in CONNECTING status.

If I restart node4, node3 will be the new leader, and now I can connected to cluster using zkCli again.

After some digging, I find node4's Leader#allowedToCommit field is false, so this cluster won't commit any new proposals.

 

I have attached a zookeeper cluster to reproduce this problem. The cluster in the attachment can run in one single machine.

 

Issue Links

• duplicates: ZOOKEEPER-3829 Zookeeper refuses request after node expansion