[ZOOKEEPER-3892] client cannot connect to server when we do a large scala benchmark test

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: server
• Assignee: Ling Mao
• Reporter: Ling Mao
• Created: 17/Jul/20 09:51
• Updated: 18/Jul/20 09:33

Description

We have three servers(s1[follower], s2[Leader], s3[follower]), after doing a large scala benchmark test, the client cannot connect to server anymore.

No any resource panic(CPU, memory)

 

# connect to s2[Leader]

2020-07-17 17:44:56,767 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.2020-07-17 17:44:56,767 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 30000ms for session id 0x0 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236)2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1159] - Opening socket connection to server localhost/127.0.0.1:2182.2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1161] - SASL config status: Will not attempt to authenticate using SASL (unknown error)2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@993] - Socket connection established, initiating session, client: /127.0.0.1:64219, server: localhost/127.0.0.1:21822020-07-17 17:45:28,522 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1235] - Client session timed out, have not heard from server in 30003ms for session id 0x02020-07-17 17:45:28,522 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 30003ms for session id 0x0 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236)2020-07-17 17:45:30,115 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1159] - Opening socket connection to server localhost/127.0.0.1:2182.2020-07-17 17:45:30,115 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1161] - SASL config status: Will not attempt to authenticate using SASL (unknown error)2020-07-17 17:45:30,116 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@993] - Socket connection established, initiating session, client: /127.0.0.1:64223, server: localhost/127.0.0.1:2182
[zk: 127.0.0.1:2182(CONNECTING) 0]

./zkServer.sh status
JMX enabled by default
Using config: /data/software/zookeeper/zookeeper-two/bin/../conf/zoo.cfg
Mode: leader

However, we cannot connect to s2,s3, but we can connect to s1 and create znodes

[zk: 127.0.0.1:2181(CONNECTED) 0] getAllChildrenNumber /
1645338
[zk: 127.0.0.1:2181(CONNECTED) 1] create /test-666
Created /test-666
[zk: 127.0.0.1:2181(CONNECTED) 2] create /test-777 "777"
Created /test-777
[zk: 127.0.0.1:2181(CONNECTED) 3] get /test-777
777
[zk: 127.0.0.1:2181(CONNECTED) 4] create /test-888 "88"
Created /test-888
[zk: 127.0.0.1:2181(CONNECTED) 5] get /test-888
88

Attach the logs for debugging