Blocker Description
Through some internal testing, we found that Ambari's use of Phoenix to host metrics data was leaking ConnectionQueryServices (and thus HConnections and ZK connections), ultimately running into ZK's rate limiting maxClientCnxns.
After a bit of digging around (and revisiting the old issues around this topic PHOENIX-3607, PHOENIX-3611, etc), I finally realized that the logic in ConnectionInfo was simply not correctly handling the _HOST special string in the principal (that UGI will replace with the FQDN for the current host).
This resulted in Phoenix repeatedly re-logging in the user when they created a new Connection instead of using the UGI current user, leaking another set of connections.