We have a setup with a network of two brokers. They're running on broker1.foo.bar.com and broker2.foo.bar.com.
We have a producer on producer.bar.com which would not failover when one of the brokers went down. We're using failover:(tcp://broker1.foo.bar.com:7250,tcp://broker2.foo.bar.com:7250) as the connect string.
We couldn't figure out why until we enabled debug logging:
[DEBUG] [2015-10-29T11:37:15,063] [ActiveMQ Task-4] Attempting 0th connect to: tcp://broker1:7250 [FailoverTransport] [DEBUG] [2015-10-29T11:37:15,068] [ActiveMQ Task-4] Connect fail to: tcp://broker1:7250, reason: java.net.UnknownHostException: broker1 [FailoverTransport]
(off topic: why is there only debug logging on failing attempts when there is info logging on successful attempts?)
Since the brokers are on foo.bar.com, and the producer is on bar.com, a hostname-only lookup fails.
Our workaround is to add foo.bar.com to the DNS search domain on producer.bar.com. (Either in /etc/resolv.conf or as a LOCALDOMAIN environment variable)