[ZOOKEEPER-3829] Zookeeper refuses request after node expansion

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.1, 3.5.8
• Fix Version/s: 3.5.9, 3.7.0, 3.6.2
• Component/s: server
• Assignee: Mate Szalay-Beko
• Reporter: Sundy Li
• Created: 15/May/20 03:12
• Updated: 10/Sep/20 10:43
• Resolved: 03/Jun/20 07:59

Description

It's easy to reproduce this bug.

//代码占位符
 
Step 1. Deploy 3 nodes  A,B,C with configuration A,B,C .
Step 2. Deploy node ` D` with configuration  `A,B,C,D` , cluster state is ok now.
Step 3. Restart nodes A,B,C with configuration A,B,C,D, then the leader will be D, cluster hangs, but it can accept `mntr` command, other command like `ls /` will be blocked.

Step 4. Restart nodes D, cluster state is back to normal now.
 

 

We have looked into the code of 3.5.6 version, and we found it may be the issue of  `workerPool` .

The `CommitProcessor` shutdown and make `workerPool` shutdown, but `workerPool` still exists. It will never work anymore, yet the cluster still thinks it's ok.

 

I think the bug may still exist in master branch.

We have tested it in our machines by reset the `workerPool` to null. If it's ok, please assign this issue to me, and then I'll create a PR. 

 

 

 

Issue Links

• fixes: ZOOKEEPER-3920 Zookeeper clients timeout after leader change due to 0.0.0.0 address when in docker environment
• is duplicated by: ZOOKEEPER-3842 Rolling scale up of zookeeper cluster does not work with reconfigEnabled=false
• is duplicated by: ZOOKEEPER-3814 ZooKeeper config propagates even with disabled dynamic reconfig
• is duplicated by: ZOOKEEPER-3830 After add a new node, zookeeper cluster won't commit any proposal if this new node is leader
• relates to: ZOOKEEPER-3920 Zookeeper clients timeout after leader change due to 0.0.0.0 address when in docker environment
• links to: Github Pull Request #1356
• links to: Github Pull Request #1370