[PHOENIX-3189] HBase/ZooKeeper connection leaks when providing principal/keytab in JDBC url

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 4.8.0
• Fix Version/s: 4.9.0, 4.8.1
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 17/Aug/16 03:43
• Updated: 26/May/17 19:02
• Resolved: 01/Sep/16 14:58

Description

We've been doing some more testing after PHOENIX-3126 and, with the help of arpitgupta and harsha_ch, we've found an issue in a test between Storm and Phoenix.

Storm was configured to create a JDBC Bolt, specifying the principal and keytab in the JDBC URL, relying on PhoenixDriver to do the Kerberos login for them. After PHOENIX-3126, a ZK server blacklisted the host running the bolt, and we observed that there were over 140 active ZK threads in the JVM.

This results in a subtle change where every time the client tries to get a new Connection, we end up getting a new UGI instance (because the ConnectionQueryServicesImpl#openConnection() always does a new login).

If users are correctly caching Connections, there isn't an issue (best as I can presently tell). However, if users rely on the getting the same connection every time (the pre-PHOENIX-3126), they will saturate their local JVM with connections and crash.

Issue Links

• is related to: PHOENIX-3232 Automatic Kerberos login via JDBC url can result in clients using other's credentials
• relates to: PHOENIX-3891 ConnectionQueryServices leak on auto-Kerberos-login without REALM in URL
• relates to: PHOENIX-3216 Kerberos ticket is not renewed when using Kerberos authentication with Phoenix JDBC driver
• links to: GitHub Pull Request #191