Affects Version/s: 3.5.7
Fix Version/s: None
Issue: Data Inconsistency Between Zookeeper Leader and zookeeper Followers. zookeeper followers and zookeeper leaders have other information. We try to delete the information from the follower's, but information not present in zookeeper leader, it's throwing error like Node does not exist:
Expected behaviour: Data consistency between zookeeper leader and Zookeeper followers should be same.
Steps followed as part of troubleshooting:
We have 5 zookeepers in clusters.
Step1: verified all zookeepers are following the leader or not?. As per below information its following all 4 zookeepers to zookeeper leader
zk_version 3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/11/2020 11:30 GMT
Step 2: Verified znode in all the zookeepers , but we are not getting same information from zookeeper leader and followers.
Step 3: Try to delete the Zookeeper node and received below error. Also, we are suspecting that when trying to delete the info of znode, it's trying to reach zookeeper leader and throwing Node does not exist error.
14:04:54.769 [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
14:04:54.775 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
14:04:54.824 [main-SendThread(188.8.131.52:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 11-111-226-146.ebiz.verizon.com/184.108.40.206:2181. Will not attempt to authenticate using SASL (unknown error)
14:04:54.831 [main-SendThread(220.127.116.11:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /18.104.22.168:38804, server: 11-111-226-146.ebiz.verizon.com/22.214.171.124:2181
14:04:54.835 [main-SendThread(126.96.36.199:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 11-111-226-146.ebiz.verizon.com/188.8.131.52:2181, sessionid = 0x500001bbbeb0651, negotiated timeout = 20000
WatchedEvent state:SyncConnected type:None path:null
Node does not exist: /namespace/$tenant/$Namespace/$zk-path