[CURATOR-638] Curator disconnect from zookeeper when IPs change

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 5.2.1
• Fix Version/s: 5.4.0
• Component/s: Client, Recipes
• Environment: Docker or Kubernetes, docker example provided

Language

Java

• Assignee: Zili Chen
• Reporter: Francis Simon
• Created: 02/May/22 19:18
• Updated: 02/Apr/23 05:16
• Resolved: 17/Jul/22 08:04

Description

Blocking usage of Zookeeper in production.   Tried testing a few versions all had the issue.  Effects any recipes that use ephemeral nodes.  Example attached.

We use multiple Apache Curator recipes in our system which is running in Docker and Kubernetes.   The behavior I am seeing is that curator appears to resolve to the IP address of the containers rather than being tied to DNS names.   I have seen old tickets on this, but the behavior is reproducible on the latest code release.    

We are running zookeeper in containers on kubernetes.  In kubernetes many things could cause a container to move hosts, the pod disruption budget ensures that a quorum is always present.   But with this bug if all nodes move for any reason and get new IP addresses clients will disconnect when they shouldn't.  Disconnecting has the bad side effect that all ephemeral nodes are lost.   This effects for us coordination, distributed locking and service discovery.   Causes production downtime so marked as a Blocker.

I have a simple sample which just uses the service discovery recipe to register a bunch of services in zookeeper.  I run the example in docker compose.   It is 100% reproducible.

 

# Standup zookeeper and wait for it to be healthy
docker-compose up -d zookeeper1 zookeeper2 zookeeper3 

# Stand up a server and make sure it is connected and working as expected
docker-compose up -d server1

# Take down a single zookeeper node and stand up another agent.
# The agent will grab the old zookeepers IP address
docker-compose rm -s zookeeper1
docker-compose up -d server2

# Bring the zookeeper node back up.  
# Wait for it be healthy
docker-compose up -d zookeeper1

# Then take down the next zookeeper node and stand up another agent.
# The agent will grab the old zookeepers IP address
docker-compose rm -s zookeeper2
docker-compose up -d server3

# Bring the zookeeper node back up. 
# Wait for it be healthy
docker-compose up -d zookeeper2

# Then take down the next zookeeper node and stand up another agent.
# The agent will grab the old zookeepers IP address
docker-compose rm -s zookeeper3 
docker-compose up -d server4 

# Bring the zookeeper node back up. 
# Wait for it be healthy 
docker-compose up -d zookeeper3

 

At the time of taking down the 3rd zookeeper node, the first server1 that was stood up will now receive a disconnected status because the IP of all three nodes has no changes form the original IP addresses.  

 

server1_1     | Query instances for servicetest
server1_1     | Exception in thread "main" java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
server1_1     | 	at org.apache.curator.shaded.com.google.common.base.Throwables.propagate(Throwables.java:241)
server1_1     | 	at org.apache.curator.utils.ExceptionAccumulator.propagate(ExceptionAccumulator.java:38)
server1_1     | 	at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:171)
server1_1     | 	at org.apache.curator.shaded.com.google.common.io.Closeables.close(Closeables.java:78)
server1_1     | 	at org.apache.curator.utils.CloseableUtils.closeQuietly(CloseableUtils.java:59)
server1_1     | 	at zkissue.App.main(App.java:72)
server1_1     | Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
server1_1     | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
server1_1     | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
server1_1     | 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:2001)
server1_1     | 	at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274)
server1_1     | 	at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268)
server1_1     | 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
server1_1     | 	at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265)
server1_1     | 	at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:249)
server1_1     | 	at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:34)
server1_1     | 	at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalUnregisterService(ServiceDiscoveryImpl.java:520)
server1_1     | 	at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:157)
server1_1     | 	... 3 more

 

 This causes it to disconnect and lose its discovery state which can be seen from the other services.

server2_1     | Query instances for servicetest
server2_1     | test
server2_1     | 	service description: http://server-4:57456
server2_1     | 	service description: http://server-3:37740
server2_1     | 	service description: http://server-2:40219

 

Should mention that the Zookeeper cluster is always happy and healthy.   This is a client side issue.

 

 

Issue Links

• is duplicated by: CURATOR-578 EnsembleTracker replace hostname connectString with wrong ip from zk config
• is related to: CURATOR-664 EnsembleTracker use resolved server address in connection string if client address is wildcard
• links to: GitHub Pull Request #425
• links to: GitHub Pull Request #452