[DRILL-6090] While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.13.0
• Component/s: Client - JDBC
• Labels: ready-to-commit

Environment

Centos 65, Java 7, Drill JDBC 1.12.0

• Flags: Patch, Important
• Assignee: Milind Takawale
• Reporter: Milind Takawale
• Reviewer: Arina Ielchiieva
• Created: 16/Jan/18 14:14
• Updated: 24/Feb/18 01:22
• Resolved: 24/Feb/18 01:22

Description

I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.

Steps to reproduce the issue

1. Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)
2. Now stop the drill-bit services or block the drill-bit IPs using iptable rules
3. Truncate catalina logs
4. Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.
5. Take thread dump using kill -QUIT <java process id>
6. grep -c "Curator-Framework-0" catalina.out

Observe that the curator framework thread just keep on accumulating

RCA:

1. ZKClusterCoordinator creates curator threads in the constructor
2. ZKClusterCoordinator is instantiated by DrillClient.connect
3. DrillClient.connect is called in DrillConnectionImpl constructor

Fix:

Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl  constructor.

 

 

 

Issue Links

• links to: GitHub Pull Request #1094