We have a setup with a network of two brokers. They're running on broker1.foo.bar.com and broker2.foo.bar.com.
We have a producer on producer.bar.com which would not failover when one of the brokers went down. We're using failover:(tcp://broker1.foo.bar.com:7250,tcp://broker2.foo.bar.com:7250) as the connect string.
We couldn't figure out why until we enabled debug logging:
(off topic: why is there only debug logging on failing attempts when there is info logging on successful attempts?)
Since the brokers are on foo.bar.com, and the producer is on bar.com, a hostname-only lookup fails.
Our workaround is to add foo.bar.com to the DNS search domain on producer.bar.com. (Either in /etc/resolv.conf or as a LOCALDOMAIN environment variable)