Storm was configured to create a JDBC Bolt, specifying the principal and keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for them. After
PHOENIX-3126, a ZK server blacklisted the host running the bolt, and we observed that there were over 140 active ZK threads in the JVM.
This results in a subtle change where every time the client tries to get a new Connection, we end up getting a new UGI instance (because the ConnectionQueryServicesImpl#openConnection() always does a new login).
If users are correctly caching Connections, there isn't an issue (best as I can presently tell). However, if users rely on the getting the same connection every time (the pre-
PHOENIX-3126), they will saturate their local JVM with connections and crash.