I have a zookeeper cluster with 3 nodes, node3 is the leader of the cluster.
With dynamic reconfiguration disabled, I scale this cluster to 4 nodes with 2 steps:
- Start node4 with new config, now node4 is a follower.
- Modify config and restart node1, node2 and node3 one by one.
The new cluster config is:
After restart, node4 is the leader of this cluster. But I cannot connect to this cluster using zkCli now, zkCli keeps in CONNECTING status.
If I restart node4, node3 will be the new leader, and now I can connected to cluster using zkCli again.
After some digging, I find node4's Leader#allowedToCommit field is false, so this cluster won't commit any new proposals.
I have attached a zookeeper cluster to reproduce this problem. The cluster in the attachment can run in one single machine.