tomcat-jdbc version: 7.0.54 In this code fragment from FairBlockingQueue::poll method Line 144 //the queue is empty we will wait for an object ExchangeCountDownLatch<E> c = new ExchangeCountDownLatch<E>(1); //add to the bottom of the wait list waiters.addLast(c); //unlock the global lock lock.unlock(); //wait for the specified timeout if (!c.await(timeout, unit)) { //if we timed out, remove ourselves from the waitlist lock.lock(); waiters.remove(c); lock.unlock(); } //return the item we received, can be null if we timed out result = c.getItem(); When the 'c.await' is interrupted, the Latch is not removed from 'waiters'. As I understand, it can hold on to the connection object. Thus the pool is leaking connections, and at some time, all new requests just hang on the c.await indefinitely. The fix is pretty simple. Catch InterruptedException (and any other non-checked exception) and do a cleanup there. boolean needsWaitersCleanup = false; try { if (!c.await(timeout, unit)) { needsWaitersCleanup = true; } } catch ( InterruptedException | RuntimeException | Error e ) { needsWaitersCleanup = true; throw e; } finally { if ( needsWaitersCleanup ) { //if we timed out, remove ourselves from the waitlist lock.lock(); try { waiters.remove(c); } finally { lock.unlock(); } } }
I've encountered this bug in production, under a heavy load. The problem here is that it's manifestation is pretty random. One workaround, until this bug is fixed, is to turn off 'fairQueue' property of PoolProperties. When 'fairQueue' is off, current implementation falls back to 'java.util.concurrent.ArrayBlockingQueue', which does not have this issue.
https://issues.apache.org/bugzilla/show_bug.cgi?id=53367 *** This bug has been marked as a duplicate of bug 53367 ***