[ZOOKEEPER-2986] My id not in the peer list

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.3

Environment

Running in a docker container in kubernetes 1.5

• Assignee: Unassigned
• Reporter: Mohammad Etemad
• Created: 22/Feb/18 19:25
• Updated: 29/Aug/19 02:25

Description

Zookeeper 3.5.3-beta is throwing the following error. I am facing the issue of  "My id 1 not in the peer list". If I use the alpha version (3.5.2) and then upgrade to the 3.5.3 beta version, the problem goes away. But if I implement the 3.5.3 version directly, the clustering never  happens and I get the error. To give you a bit more overview of the implementation:
 
The pods use a persistent volume claim on a gluster volume. Each pod is assigned its own volume on the gluster file system. I run zookeeper as a stateful set with 3 pods. 
 
In my cfg file I have:
 

standaloneEnabled=false 
tickTime=2000 
initLimit=10 
syncLimit=5 
#snapshot file dir 
dataDir=/data 
#tran log dir 
dataLogDir=/dataLog 
#zk log dir 
logDir=/logs 
4lw.commands.whitelist=* 
dynamicConfigFile=/opt/zookeeper/conf/zoo_replicated1.cfg.dynamic

  
and in my cfg.dynamic file I have:
  

server.0=zookeeper-0:2888:3888 
server.1=zookeeper-1:2888:3888 
server.2=zookeeper-2:2888:3888

  
Has there been any change on the clustering side of things that makes the new version not work?
Sample logs:

2018-02-22 19:21:18,078 [myid:1] - ERROR [main:QuorumPeerMain@98] - Unexpected exception, exiting abnormally
 java.lang.RuntimeException: My id 1 not in the peer list
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:770)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:185)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:120)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)