Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3892

client cannot connect to server when we do a large scala benchmark test

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: server
    • Labels:
      None

      Description

      We have three servers(s1[follower], s2[Leader], s3[follower]), after doing a large scala benchmark test, the client cannot connect to server anymore.

      No any resource panic(CPU, memory)

       

      # connect to s2[Leader]
      
      2020-07-17 17:44:56,767 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.2020-07-17 17:44:56,767 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 30000ms for session id 0x0 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236)2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1159] - Opening socket connection to server localhost/127.0.0.1:2182.2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1161] - SASL config status: Will not attempt to authenticate using SASL (unknown error)2020-07-17 17:44:58,519 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@993] - Socket connection established, initiating session, client: /127.0.0.1:64219, server: localhost/127.0.0.1:21822020-07-17 17:45:28,522 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1235] - Client session timed out, have not heard from server in 30003ms for session id 0x02020-07-17 17:45:28,522 [myid:127.0.0.1:2182] - WARN  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1278] - Session 0x0 for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 30003ms for session id 0x0 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1236)2020-07-17 17:45:30,115 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1159] - Opening socket connection to server localhost/127.0.0.1:2182.2020-07-17 17:45:30,115 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1161] - SASL config status: Will not attempt to authenticate using SASL (unknown error)2020-07-17 17:45:30,116 [myid:127.0.0.1:2182] - INFO  [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@993] - Socket connection established, initiating session, client: /127.0.0.1:64223, server: localhost/127.0.0.1:2182
      [zk: 127.0.0.1:2182(CONNECTING) 0]
      
      ./zkServer.sh status
      JMX enabled by default
      Using config: /data/software/zookeeper/zookeeper-two/bin/../conf/zoo.cfg
      Mode: leader
      

      However, we cannot connect to s2,s3, but we can connect to s1 and create znodes

      [zk: 127.0.0.1:2181(CONNECTED) 0] getAllChildrenNumber /
      1645338
      [zk: 127.0.0.1:2181(CONNECTED) 1] create /test-666
      Created /test-666
      [zk: 127.0.0.1:2181(CONNECTED) 2] create /test-777 "777"
      Created /test-777
      [zk: 127.0.0.1:2181(CONNECTED) 3] get /test-777
      777
      [zk: 127.0.0.1:2181(CONNECTED) 4] create /test-888 "88"
      Created /test-888
      [zk: 127.0.0.1:2181(CONNECTED) 5] get /test-888
      88
      

      Attach the logs for debugging

       

       

       

        Attachments

        1. zookeeper-3.out
          3.06 MB
          Ling Mao
        2. zookeeper-2.out
          988 kB
          Ling Mao
        3. zookeeper-1.out
          751 kB
          Ling Mao
        4. zoo.cfg
          0.5 kB
          Ling Mao

          Activity

            People

            • Assignee:
              maoling Ling Mao
              Reporter:
              maoling Ling Mao

              Dates

              • Created:
                Updated:

                Issue deployment