[ZOOKEEPER-4728] Zookeepr cannot bind to itself forever if DNS is not ready when startup

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.4
• Fix Version/s: 3.10.0, 3.8.4, 3.9.2
• Labels: pull-request-available
• Assignee: Luke Chen
• Reporter: Luke Chen
• Created: 27/Jul/23 12:03
• Updated: 18/Oct/24 16:59
• Resolved: 29/Aug/23 10:46

Description

Note: This issue also happened in the latest `master` branch

 

When the leader tried to bind the host/IP to get connection from followers, if the DNS is not ready at first, it'll always stay in <unresolved> state forever. The error log is like this:

 

2023-07-26 00:25:25,251 ERROR Couldn't bind to localhost1/<unresolved>:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeer[myid=1]]java.net.SocketException: Unresolved address    at java.base/java.net.ServerSocket.bind(ServerSocket.java:380)    at java.base/java.net.ServerSocket.bind(ServerSocket.java:342)    at org.apache.zookeeper.server.quorum.Leader.createServerSocket(Leader.java:315)    at org.apache.zookeeper.server.quorum.Leader.lambda$new$0(Leader.java:294)    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)    at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573)    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)    at org.apache.zookeeper.server.quorum.Leader.<init>(Leader.java:297)    at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1272)    at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1479)2023-07-26 00:25:25,252 WARN Unexpected exception (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeer[myid=1]]java.io.IOException: Leader failed to initialize any of the following sockets: [localhost1/<unresolved>:2888]    at org.apache.zookeeper.server.quorum.Leader.<init>(Leader.java:300)    at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1272)    at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1479) 

 

 

This repeatedly appear and never successfully bind to the address, so the quorum never formed.

 

Reproduce steps:

1. setup 1 zookeeper node, and set the zookeeper connection config as:

server.1=localhost1:2888:3888

Note, it's "localhost1"

2. startup the zookeeper node, it'll show the `Exception while listening` error , as well as the `Couldn't bind to localhost1/<unresolved>:2888 ` error like above. This is to simulate the DNS is not ready when zookeeper startup. It's quite common in k8s environment.

3. edit /etc/hosts, map `localhost1` into `127.0.0.1`

4. You can see the log, the `Exception while listening` error is gone, but `Couldn't bind to localhost1/<unresolved>:2888 ` still keeps appearing, and the quorum never formed.

 

Note: The `Exception while listening` can be self-healing is because it re-resolve the hostname each time it tried to bind the hostname. So we should apply the same solution to the leader binding. (i.e. ZOOKEEPER-3991)

Issue Links

• is related to: ZOOKEEPER-3940 Zookeeper restart of leader causes all zk nodes to not serve requests
• relates to: ZOOKEEPER-3991 QuorumCnxManager Listener port bind retry does not retry DNS lookup
• links to: GitHub Pull Request #2040