[ZOOKEEPER-3776] Cluster stuck not forming up quorum - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Duplicate
Affects Version/s: None
Fix Version/s: 3.6.1
Component/s: None
Labels:
None

Description

We had working Zookeeper 3.4.8 that came with Kafka. 3 machines cluster. Now I'm trying to upgrade it to 3.6.0. I didn't touch the data directory, just set up 3.6.0 Zookeeper in the new directory and updated config. Stopped all Zookeepers. Now I've started eache Zk in turn with some delay. And they are not forming up quorum, stuck seemingly forever, sometimes trying to elect and sometimes throwing nonsensical errors.

For example, on node 1 (sorry don't know how to format this properly in jira):

Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,133 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,134 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,335 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer state changed: following
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1453] - FOLLOWING
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246] - minSessionTimeout set to 6000
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255] - maxSessionTimeout set to 60000
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] - Response cache size is initialized with value 400.
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111] - zookeeper.pathStats.slotCapacity = 60
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112] - zookeeper.pathStats.slotDuration = 15
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113] - zookeeper.pathStats.maxDepth = 6
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,536 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114] - zookeeper.pathStats.initialDelay = 5
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115] - zookeeper.pathStats.delay = 5
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116] - zookeeper.pathStats.enabled = false
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470] - The max bytes for all large requests are set to 104857600
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484] - The large request threshold is set to -1
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir /kafka/kafka_zookeeper_data/logdir/version-2 snapdir /kafka/kafka_zookeeper_data/version-2
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@75] - FOLLOWING - LEADER ELECTION TOOK - 92790 MS
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,537 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: following - discovery
Mar 31 20:05:21 java[24055]: 2020-03-31 20:05:21,736 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,538 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:FOLLOWING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@129] - Exception when following the leader
Mar 31 20:05:22 java[24055]: java.lang.IllegalArgumentException
Mar 31 20:05:22 java[24055]: at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1307)
Mar 31 20:05:22 java[24055]: at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1195)
Mar 31 20:05:22 java[24055]: at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89)
Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:275)
Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87)
Mar 31 20:05:22 java[24055]: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1455)
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@292] - shutdown Follower
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: looking
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - WARN [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] - PeerState set to LOOKING
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] - LOOKING
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931] - New election. My id = 1, proposed zxid=0x10002b653
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,539 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,542 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,544 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,744 [myid:1] - INFO [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 400
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,745 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[24055]: 2020-03-31 20:05:22,746 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0

----------
On node 2 in the same time:

Mar 31 20:04:39 java[23556]: 2020-03-31 20:04:39,764 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,130 [myid:2] - WARN [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1471] - Unexpected exception
Mar 31 20:05:21 java[23556]: java.lang.InterruptedException: Timeout while waiting for epoch from quorum
Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader.getEpochToPropose(Leader.java:1425)
Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:599)
Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1468)
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@792] - Shutting down
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@798] - Shutdown called. For the reason Forcing shutdown
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: looking
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1501] - PeerState set to LOOKING
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1371] - LOOKING
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - WARN [LearnerCnxAcceptorHandler-/172.30.2.252:2888:Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler@523] - Exception while shutting down acceptor.
Mar 31 20:05:21 java[23556]: java.net.SocketException: Socket closed
Mar 31 20:05:21 java[23556]: at java.net.PlainSocketImpl.socketAccept(Native Method)
Mar 31 20:05:21 java[23556]: at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
Mar 31 20:05:21 java[23556]: at java.net.ServerSocket.implAccept(ServerSocket.java:545)
Mar 31 20:05:21 java[23556]: at java.net.ServerSocket.accept(ServerSocket.java:513)
Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.acceptConnections(Leader.java:510)
Mar 31 20:05:21 java[23556]: at org.apache.zookeeper.server.quorum.Leader$LearnerCnxAcceptor$LearnerCnxAcceptorHandler.run(Leader.java:493)
Mar 31 20:05:21 java[23556]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Mar 31 20:05:21 java[23556]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Mar 31 20:05:21 java[23556]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Mar 31 20:05:21 java[23556]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Mar 31 20:05:21 java[23556]: at java.lang.Thread.run(Thread.java:745)
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,131 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@931] - New election. My id = 2, proposed zxid=0x10002b653
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,133 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,134 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 400
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,335 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,336 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 800
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,736 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[23556]: 2020-03-31 20:05:21,737 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FastLeaderElection@966] - Notification time out: 1600
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,538 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LOOKING; n.sid:1, n.state:FOLLOWING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@857] - Peer state changed: leading
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@1465] - LEADING
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1246] - minSessionTimeout set to 6000
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1255] - maxSessionTimeout set to 60000
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ResponseCache@45] - Response cache size is initialized with value 400.
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@111] - zookeeper.pathStats.slotCapacity = 60
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@112] - zookeeper.pathStats.slotDuration = 15
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,539 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@113] - zookeeper.pathStats.maxDepth = 6
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,540 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@114] - zookeeper.pathStats.initialDelay = 5
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@115] - zookeeper.pathStats.delay = 5
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):RequestPathMetricsCollector@116] - zookeeper.pathStats.enabled = false
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1470] - The max bytes for all large requests are set to 104857600
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@1484] - The large request threshold is set to -1
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@329] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir /kafka/kafka_zookeeper_data/logdir/version-2 snapdir /kafka/kafka_zookeeper_data/version-2
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Leader@581] - LEADING - LEADER ELECTION TOOK - 1410 MS
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):QuorumPeer@863] - Peer state changed: leading - discovery
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,541 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):FileTxnSnapLog@470] - Snapshotting: 0x10002b653 to /kafka/kafka_zookeeper_data/version-2/snapshot.10002b653
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LEADING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,544 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:3, n.state:LEADING, n.leader:3, n.round:0x4a, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[23556]: 2020-03-31 20:05:22,568 [myid:2] - INFO [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):ZooKeeperServer@519] - Snapshot taken in 26 ms

----------------------
node 3:

Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,133 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,134 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,335 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:21 java[20867]: 2020-03-31 20:05:21,736 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,540 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,544 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:2, n.state:LOOKING, n.leader:2, n.round:0x4b, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:22 java[20867]: 2020-03-31 20:05:22,745 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0
Mar 31 20:05:23 java[20867]: 2020-03-31 20:05:23,147 [myid:3] - INFO [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@376] - Notification: my state:LEADING; n.sid:1, n.state:LOOKING, n.leader:1, n.round:0x4c, n.peerEpoch:0x1, n.zxid:0x10002b653, message format version:0x2, n.config version:0x0

Attachments

Issue Links

duplicates

ZOOKEEPER-3758 Update from 3.5.7 to 3.6.0 does not work

Resolved

Activity

People

Assignee:: Mate Szalay-Beko

Reporter:: Philipp

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 31/Mar/20 20:13

Updated:: 02/Apr/20 17:00

Resolved:: 02/Apr/20 16:53