Details
Description
I've noticed an issue with Curator 3.2.1 which relates to the fix from CURATOR-345 (also reported by me).
When we would reconnect after losing connection to Zookeeper (due to network issues), our services would always have the wrong connection string, and never manage to reconnect to the Zookeeper cluster. Assuming that 10.1.2.3 is our zookeeper server, and we have two scenarios (with different zoo.cfg files) we were seeing the following results when a reconnection was established:
Scenario 1: ClientCnxn - Opening socket connection to server 0.0.0.0/0.0.0.0:2181.
Scenario 2: ClientCnxn - Opening socket connection to server 10.1.2.3/10.1.2.3:2888.
Obviously these are both undesirable connection strings, as both are wrong. The issue arises in the EnsembleTracker.processConfigData() when we reconnect to Zookeeper. The config coming from zookeeper is in the format:
server.<positive id> = <address1>:<port1>:<port2>[:role];[<client port address>:]<client port>
As we can see, both [:role] and [<client port address>:] are optional. Hence, the following string is perfectly valid:
server.1=10.1.2.3:2888:3888:participant;2181
When Zookeeper sends this, it defaults the clientAddress to 0.0.0.0, so we retrieve the following value in EnsembleTracker:
server.1=10.1.2.3:2888:3888:participant;0.0.0.0:2181
The resulting connection string, therefore, turns in to 0.0.0.0:2181 instead of 10.1.2.3:2181, and Curator creates a new ZooKeeper to connect to that IP - which obviously never works.
In the second scenario, our connection string looks a bit different. It is wrong according to the docs, but is valid:
server.1=10.1.2.3:2888:3888:participant
Now, this is missing the client port and address. That means that the resulting string from the EnsembleTracker is 10.1.2.3:2888 - which isn't desired. Including the port would just lead to the above scenario.
From what I can see, the EnsembleTracker.configToConnectionString() method is the issue here:
InetSocketAddress address = Objects.firstNonNull(server.clientAddr, server.addr);
sb.append(address.getAddress().getHostAddress()).append(":").append(address.getPort());
In the above cases, both the server.Addr and server.clientAddr values are wrong. We also prefer the value of clientAddr for some reason, which doesn't look right to me (given that it can be 0.0.0.0 or 127.0.0.1).
It seems to me that Curator should use server.Addr.getHostAddress() with server.clientAddr.getPort(). When the clientAddr is missing, however, I'm not sure what should be done.
Attachments
Issue Links
- supercedes
-
CURATOR-402 Curator should not use QuorumServer.addr when updating the connect string dynamically
-
- Closed
-
- links to
