Affects Version/s: 2.1.0
Fix Version/s: None
I'm running Kafka on AWS autoscaling group. Currently we are having infrastructure immutable and we have no fixed IP addresses.
When we recreate out Kafka cluster all brokers get new IP addresses, even in case when all nodes die horrible death simultaneously.
Looking at the code IP addresses are resolved and added ClusterConnectionStates class line 119:
nodeState.put(id, new NodeConnectionState(ConnectionState.CONNECTING, now,
this.reconnectBackoffInitMs, ClientUtils.resolve(host, clientDnsLookup)));
At this point ClientUtils.resolve resolves IP addreses. If brokers receive new IP addresses each subsequent metadata fetch will fail because none of the nodes will be addressable by IP addresses that where resolved on previous metadata fetch.
Since addresses list will never be evicted old IP addresses will stay there forever.