[ZOOKEEPER-2343] Zookeeper 3.5.1 failed to deploy into the kubernetes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.1
• Fix Version/s: 3.5.1
• Component/s: quorum

Environment

CentOS Linux release 7.1.1503 (Core)
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
zookeeper: 3.5.1-alpha-1693007

• Tags: dynamic reconfiguration
• Flags: Important
• Assignee: Unassigned
• Reporter: cheyang
• Created: 15/Dec/15 11:06
• Updated: 30/Nov/17 09:27

Description

I'd like to setup 3-nodes zookeeper cluster with version 3.5.1. Because the network model of kubernetes is that pod and service has different ip address. In order to deploy it into kubernetes, I have to make zookeeper pods itself to be 0.0.0.0, so it can be started correctly. the configuration as below:

zk1: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=0.0.0.0:2888:3888:participant;2181
server.2=10.62.56.192:2888:3888:participant;2181
server.3=10.62.56.193:2888:3888:participant;2181

zk2: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=10.62.56.191:2888:3888:participant;2181
server.2=0.0.0.0:2888:3888:participant;2181
server.3=10.62.56.193:2888:3888:participant;2181

zk3: zoo.cfg

standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic

zoo.cfg.dynamic

server.1=10.62.56.191:2888:3888:participant;2181
server.2=10.62.56.192:2888:3888:participant;2181
server.3=0.0.0.0:2888:3888:participant;218

The result is that:
1. Looks like the election is successful. a new dynamic file is generated in every node:/opt/zookeeper/conf/zoo.cfg.dynamic.100000000
like below
server.1=10.62.56.191:2888:3888:participant;0.0.0.0:2181
server.2=0.0.0.0:2888:3888:participant;0.0.0.0:2181
server.3=10.62.56.193:2888:3888:participant;0.0.0.0:2181

2. But the cluster doesn't really work, I saw the errors:

0:0:2181)(secure=disabled):Learner@273] - Unexpected exception, tries=3, remaining init limit=16997, connecting to /0.0.0.0:2888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.Learner.sockConnect(Learner.java:227)
at org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:256)
at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:74)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1064)
2015-12-15 04:35:00,403 [myid:1] - INFO
2015-12-15 04:35:00,585 [myid:1] - INFO [QuorumPeer[myid=1](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@198] - shutdown called
java.lang.Exception: shutdown Follower
at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1068)