Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
In our drill automation system, a batch of over 200 tests were submitted via the JDBC connection which ended in large quantities of failures. The following investigating analyses were performed to root cause the problem:
1. I put a sleep of 15 minutes in a single test case execution, and the zookeeper connection was not closed until the test terminated after sleep time ran out.
2. I then ran a couple of tests and reduced the sleep time to 5 minutes. Two zookeeper connections were established (the second one after the second test case was running) and were not closed until both tests terminated after sleep time ran out.
3. I ran a batch of four tests and further reduced the sleep time to 2 minutes. Four zookeeper connections were established (one after another till the last test case started to run) and again, none of them was closed until all tests terminated after sleep time out. Then it took about 10-15 seconds for all connections to close.
From these experiments, it appears to me that zookeeper connections are eventually closed, but not before all tests executed in one batch are completed. This explains that, with the default maximum number of zookeeper connections allowed, over 200 queries submitted in a batch are destined to fail shortly into the execution when all zookeeper connections are used up.