[ZOOKEEPER-3099] ZooKeeper cluster is unavailable for session_timeout time due to network partition in a three-node environment.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.11, 3.5.4, 3.4.12, 3.4.13
• Component/s: c client, java client
• Assignee: Unassigned
• Reporter: Jiafu Jiang
• Created: 23/Jul/18 05:30
• Updated: 15/Oct/18 01:52

Description

 

The default readTimeout timeout of ZooKeeper client is 2/3 * session_time, the default connectTimeout is session_time/hostProvider.size(). If the ZooKeeper cluster has 3 nodes, then connectTimeout is 1/3 * session_time.

 

Supports we have three ZooKeeper servers: zk1, zk2, zk3 deployed. And zk3 is now the leader. Client c1 is now connected to zk2(follower). Then we shutdown the network of zk3(leader), the same time, client c1 begin to write some data to ZooKeeper. After a (syncLimit * tick) timeout, zk2 will disconnect with leader and begin a new election, and zk2 becomes the leader.

 

The write operation will not succeed due to the leader is unavailable. It will take at most readTimeout time for c1 to discover the failure, and client c1 will try to choose another ZooKeeper server. Unfortunately, c1 may choose zk3, which is unreachable now, then c1 will spend connectTimeout to find out that zk3 is unused. Notice that readTimeout + connectTimeout = sesstion_timeout in my case(three-node cluster).

 

Therefore, in this case, the ZooKeeper cluster is unavailable for session timeout time when only one ZooKeeper server is unreachable due to network partition.

 

I have some suggestions:

1. The HostProvider used by ZooKeeper can be specified by an argument.
2. readTimeout can also be specified in any way.