Initial set of collectors is configured in the configuration files
Find available collectors by connecting to zookeeper thereafter
Remember available collectors, refresh this information only when one collector cannot be reached with a very low frequency of checks, example: random interval between 10-12 minutes, check if a new collector is available. Set a low client side zk timeout.
Round robin the write between the collector choosing the first one at random
If a write timed out, choose the next available collector, remember the attempts with the first one
Set a configurable attempt count for failed connector (default = 3), after which the failed connector is no longer in the available collectors list.
The next retry will be triggered after refresh with zookeeper is successful
If no failed collectors available, zk refresh interval should be chosen randomly between 1-2 minutes.