Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3906

Data Inconsistency Between Zookeeper Leader and zookeeper Followers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.5.7
    • Fix Version/s: None
    • Component/s: quorum
    • Labels:
      None

      Description

      Issue: Data Inconsistency Between Zookeeper Leader and zookeeper Followers. zookeeper followers and zookeeper leaders have other information. We try to delete the information from the follower's, but information not present in zookeeper leader, it's throwing error like Node does not exist:

      Expected behaviour: Data consistency between zookeeper leader and Zookeeper followers should be same.

       

      Steps followed as part of troubleshooting:

      We have 5 zookeepers in clusters.

      Step1:  verified all zookeepers are following the leader or not?. As per below information its following all 4 zookeepers to zookeeper leader

      zk_version      3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/11/2020 11:30 GMT
      zk_avg_latency  0
      zk_max_latency  823
      zk_min_latency  0
      zk_packets_received     30214264
      zk_packets_sent 32424272
      zk_num_alive_connections        7
      zk_outstanding_requests 0
      zk_server_state leader
      zk_znode_count  75190
      zk_watch_count  21394
      zk_ephemerals_count     793
      zk_approximate_data_size        24706628
      zk_open_file_descriptor_count   281
      zk_max_file_descriptor_count    4096
      zk_followers    4
      zk_synced_followers     4
      zk_pending_syncs        0
      zk_last_proposal_size   166
      zk_max_proposal_size    121947
      zk_min_proposal_size    32

      Step 2: Verified znode in all the zookeepers , but we are not getting same information from zookeeper leader and followers.

      Step 3: Try to delete the Zookeeper node and received below error. Also, we are suspecting that when trying to delete the info of znode, it's trying to reach zookeeper leader and throwing Node does not exist error. 

      Error:
      14:04:54.769 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
      14:04:54.775 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
      14:04:54.824 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181. Will not attempt to authenticate using SASL (unknown error)
      14:04:54.831 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /11.111.225.75:38804, server: 11-111-226-146.ebiz.verizon.com/11.111.20.146:2181
      14:04:54.835 [main-SendThread(11.111.226.146:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 11-111-226-146.ebiz.verizon.com/11.111.226.146:2181, sessionid = 0x500001bbbeb0651, negotiated timeout = 20000

      WATCHER::

      WatchedEvent state:SyncConnected type:None path:null
      Node does not exist: /namespace/$tenant/$Namespace/$zk-path

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gangasai42642 Gangadhar
            • Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: