Major Description
We've observed the following situation inside of a RegionServer which leaves an HConnection in a broken state as a result of the ZooKeeper client having received an AUTH_FAILED case in the Phoenix secondary indexing code-path. The result was that the HConnection used to write the secondary index updates failed every time the client re-attempted the write but we had no outward signs from the HConnection that there was a problem with that HConnection instance.
ZooKeeper programmer docs tell us that if a ZooKeeper instance goes to the AUTH_FAILED state that we must open a new ZooKeeper instance: https://zookeeper.apache.org/doc/r3.4.13/zookeeperProgrammers.html#ch_zkSessions
When a new HConnection (or one without a cached meta location) tries to access ZooKeeper to find meta's location or the cluster ID, this spin indefinitely because we can never access ZooKeeper because our client is broken from the AUTH_FAILED. For the Phoenix use-case (where we're trying to use this HConnection within the RS), this breaks things pretty fast.
The circumstances that caused us to observe this are not an HBase (or Phoenix or ZooKeeper) problem. The AUTH_FAILED exception we see is a result of networking issues on a user's system. Despite this, we can make our handling of this situation better.
We already have logic inside of RecoverableZooKeeper to re-create a ZooKeeper object when we need one (e.g. session expired/closed). We can extend this same logic to also re-create the ZK client object if we observe an AUTH_FAILED state.