[ZOOKEEPER-2480] Exhibitor and zookeeper, Cannot open channel to x at election address

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 3.4.8
• Component/s: quorum
• Assignee: Unassigned
• Reporter: Kerem Yazici
• Created: 19/Jul/16 11:11
• Updated: 19/Jul/16 15:45
• Resolved: 19/Jul/16 15:45

Description

We have 5 node ensemble setup in our tst environment and we are seeing the error below on two of the nodes. We have 3 nodes on the first data centre and 2 nodes on the second data centre and all of them are managed by exhbitor.

The problem is that the nodes from the same data centres cannot talk to the leader node on the same data centre but can talk to the other nodes without any issue.

If I bounce the leader node to force another node to get elected then the other nodes that are on the same data centre start throwing the below exception.

I'm sure the problem is with the dns name resolution but I would like to understand how zookeeper resolves these dns names and what might be the issue here so I can go back to our unix team and get this fixed.

2016-07-19 10:48:54,711 [myid:4] - WARN  [WorkerSender[myid=4]:QuorumCnxManager@400] - Cannot open channel to 5 at election address server1.dns.name/192.168.1.3:4882
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(Fast       LeaderElection.java:452)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLead       erElection.java:433)
        at java.lang.Thread.run(Thread.java:745)

zoo.cfg

#Auto-generated by Exhibitor - Fri Jul 15 11:30:52 BST 2016
#Fri Jul 15 11:30:52 BST 2016
server.2=server2.dns.name\:4881\:4882\:observer
autopurge.purgeInterval=4
server.1=server1.dns.name\:4881\:4882
initLimit=50
syncLimit=2
clientPort=4880
tickTime=2001
server.5=server5.dns.name\:4881\:4882
dataDir=/opt/app/datafabric/data/zookeeper
server.4=server4.dns.name\:4881\:4882
dataLogDir=/path/to/datalogdir
server.3=server3.dns.name\:4881\:4882
~