[ZOOKEEPER-2184] Zookeeper Client should re-resolve hosts when connection attempts fail

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.4.6, 3.4.7, 3.4.8, 3.4.9, 3.4.10, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.4.11
• Fix Version/s: 3.6.0, 3.4.13, 3.5.5
• Component/s: java client
• Labels: easyfix patch pull-request-available

Environment

Ubuntu 14.04 host, Docker containers for Zookeeper & Kafka

• Assignee: Andor Molnar
• Reporter: Robert P. Thille
• Created: 07/May/15 23:46
• Updated: 03/Apr/20 22:45
• Resolved: 22/Jun/18 22:11

Description

Testing in a Docker environment with a single Kafka instance using a single Zookeeper instance. Restarting the Zookeeper container will cause it to receive a new IP address. Kafka will never be able to reconnect to Zookeeper and will hang indefinitely. Updating DNS or /etc/hosts with the new IP address will not help the client to reconnect as the zookeeper/client/StaticHostProvider resolves the connection string hosts at creation time and never re-resolves.

A solution would be for the client to notice that connection attempts fail and attempt to re-resolve the hostnames in the connectString.

Issue Links

• causes: KAFKA-4041 kafka unable to reconnect to zookeeper behind an ELB
• duplicates: ZOOKEEPER-338 zk hosts should be resolved periodically for loadbalancing amongst zk servers.
• fixes: SOLR-8868 SolrCloud: if zookeeper loses and then regains a quorum, Solr nodes and SolrJ Client do not recover and need to be restarted
• is related to: ZOOKEEPER-1666 Avoid Reverse DNS lookup if the hostname in connection string is literal IP address.
• relates to: MESOS-9113 Update vendored zookeeper to 3.4.13
• relates to: ZOOKEEPER-1506 Re-try DNS hostname -> IP resolution if node connection fails
• links to: GitHub Pull Request #150
• links to: GitHub Pull Request #199
• links to: GitHub Pull Request #199
• links to: GitHub Pull Request #451
• links to: GitHub Pull Request #534