Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2848

Getting this error in ZK server logs : shutdown Leader! reason: Not sufficient followers synced, only synced with sids

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Not A Bug
    • 3.5.1
    • None
    • server

    Description

      Hi
      I am seeing below error in ZK logs :
      "Unexpected exception causing shutdown while sock still open java.io.EOFException"
      and then ZK server shuts down with "shutdown Leader! reason: Not sufficient followers synced, only synced with sids: " error.

      I am using zookeeper-3.5.1-alpha version.
      it is ensemble of 2 servers
      Could you please help me resolve this issue
      Thanks

      config

      initLimit=10
      syncLimit=5
      maxClientCnxns=0
      tickTime=2000
      quorumListenOnAllIPs=true
      dataDir=/var/run/zookeeper/conf/default
      admin.enableServer=false
      standaloneEnabled=false
      
      

      zookeeper server logs

      114829 2017-06-22 11:24:18,182 [myid:2147483652] - INFO  [ProcessThread(sid:2147483652 cport:-1)::PrepRequestProcessor@649] - Processed session termination for sessionid:                  0x40000007cef003d
      114830 2017-06-22 11:24:18,182 [myid:2147483652] - INFO  [NIOWorkerThread-8:MBeanRegistry@119] - Unregister MBean [org.apache.ZooKeeperService:name0=ReplicatedServer_id2147483652,         name1=replica.2147483652,name2=Leader,name3=Connections,name4=128.0.0.5,name5=0x40000007cef003d]
      114831 2017-06-22 11:24:18,183 [myid:2147483652] - INFO  [NIOWorkerThread-8:NIOServerCnxn@606] - Closed socket connection for client /128.0.0.5:34651 which had sessionid                   0x40000007cef003d
      114832 2017-06-22 11:24:18,421 [myid:2147483652] - ERROR [LearnerHandler-/128.0.0.5:33610:LearnerHandler@604] - Unexpected exception causing shutdown while sock still open
      114833 java.io.EOFException
      114834     at java.io.DataInputStream.readInt(DataInputStream.java:403)
      114835     at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
      114836     at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
      114837     at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
      114838     at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:515)
      114839 2017-06-22 11:24:18,422 [myid:2147483652] - WARN  [LearnerHandler-/128.0.0.5:33610:LearnerHandler@619] - ******* GOODBYE /128.0.0.5:33610 ********
      114840 2017-06-22 11:24:18,422 [myid:2147483652] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:61808:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /        128.0.0.4:42854
      114841 2017-06-22 11:24:18,422 [myid:2147483652] - INFO  [NIOWorkerThread-4:ZooKeeperServer@969] - Client attempting to renew session 0x40000007cef0001 at /128.0.0.4:42854
      114842 2017-06-22 11:24:18,422 [myid:2147483652] - INFO  [NIOWorkerThread-4:ZooKeeperServer@678] - Established session 0x40000007cef0001 with negotiated timeout 20000 for client /         128.0.0.4:42854
      114843 2017-06-22 11:24:18,423 [myid:2147483652] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:61808:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /        128.0.0.4:42862
      114844 2017-06-22 11:24:18,423 [myid:2147483652] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:61808:NIOSe
       
      

      After some time ..

       
      114851 2017-06-22 11:24:18,423 [myid:2147483652] - INFO  [NIOWorkerThread-12:ZooKeeperServer@678] - Established session 0x40000007cef0003 with negotiated timeout 20000 for client /        128.0.0.4:42866
      114852 2017-06-22 11:24:19,001 [myid:2147483652] - INFO  [NIOServerCxnFactory.AcceptThread:/0.0.0.0:61808:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /        128.0.0.4:42892
      114853 2017-06-22 11:24:19,001 [myid:2147483652] - INFO  [NIOWorkerThread-13:ZooKeeperServer@964] - Client attempting to establish new session at /128.0.0.4:42892
      114854 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [SessionTracker:ZooKeeperServer@384] - Expiring session 0x40000007cef016c, timeout of 20000ms exceeded
      114855 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [SessionTracker:QuorumZooKeeperServer@132] - Submitting global closeSession request for session 0x40000007cef016c
      114856 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [SessionTracker:ZooKeeperServer@384] - Expiring session 0x40000007cef016d, timeout of 20000ms exceeded
      114857 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [ProcessThread(sid:2147483652 cport:-1)::PrepRequestProcessor@649] - Processed session termination for sessionid:                  0x40000007cef016c
      114858 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [SessionTracker:QuorumZooKeeperServer@132] - Submitting global closeSession request for session 0x40000007cef016d
      114859 2017-06-22 11:24:19,211 [myid:2147483652] - INFO  [ProcessThread(sid:2147483652 cport:-1)::PrepRequestProcessor@649] - Processed session termination for sessionid:                  0x40000007cef016d
      114860 2017-06-22 11:24:19,579 [myid:2147483652] - INFO  [QuorumPeer[myid=2147483652](plain=/0:0:0:0:0:0:0:0:61808)(secure=disabled):Leader@613] - Shutting down
      114861 2017-06-22 11:24:19,579 [myid:2147483652] - INFO  [QuorumPeer[myid=2147483652](plain=/0:0:0:0:0:0:0:0:61808)(secure=disabled):Leader@619] - Shutdown called
      114862 java.lang.Exception: shutdown Leader! reason: Not sufficient followers synced, only synced with sids: [ [2147483652] ]
      114863     at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:619)
      114864     at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:590)
      114865     at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1077)
      114866 2017-06-22 11:24:19,579 [myid:2147483652] - INFO  [QuorumPeer[myid=2147483652](plain=/0:0:0:0:0:0:0:0:61808)(secure=disabled):MBeanRegistry@119] - Unregister MBean [org.            apache.ZooKeeperService:name0=ReplicatedServer_id2147483652,name1=replica.2147483652,name2=Leader,name3=Connections,name4=128.0.0.4,name5=0x40000007cef006b]
      114867 2017-06-22 11:24:19,579 [myid:2147483652] - INFO  [LearnerCnxAcceptor-0.0.0.0/0.0.0.0:61809:Leader$LearnerCnxAcceptor@373] - exception while shutting down acceptor: java.net.       SocketException: Socket closed
      114868 2017-06-22 11:24:19,581 [myid:2147483652] - INFO  [QuorumPeer[myid=2147483652](plain=/0:0:0:0:0:0:0:0:61808)(secure=disabled):NIOServerCnxn@606] - Closed socket connection          for client /128.0.0.4:41674 which had sessionid 0x40000007cef006b
      
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hanm Michael Han
            cdprashanth prashantkumar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment