[DBCP-595] Connection pool can be exhausted when connections are killed on the DB side

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.11.0
• Labels: deadlock robustness

Language

java

• Assignee: Unassigned
• Reporter: Dénes Bodó
• Created: 08/Feb/24 13:26
• Updated: 29/Feb/24 18:53
• Resolved: 20/Feb/24 21:58

Description

Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool 1.5.4. These are ancient versions, I know.

Description

The issue is that when due to some network issues or "maintenance work" on the DB side (especially PostgreSQL) which causes the DB connection to be closed, it results exhausted Pool on the client side. Many threads are waiting at this point:

"pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x00007faf7903b800 nid=0x8603 waiting on condition [0x000000030f3e7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000066aca8e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324) 

According to my observation this is because the JDBC driver does not get closed on the client side, nor the abstract DBCP connection org.apache.commons.dbcp2.PoolableConnection .

Repro

(Un)Fortunately I can reproduce the issue using the latest and greatest commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.

I've just created a Java application to reproduce the issue: https://github.com/dionusos/pool_exhausted_repro . See README.md for detailed repro steps.

Kind of solution?

To be honest I am not really familiar with DBCP but with this change I managed to make my application more robust:

diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
index 440cb756..678550bf 100644
--- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
+++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
@@ -214,6 +214,10 @@ public class PoolableConnection extends DelegatingConnection<Connection> impleme
     @Override
     protected void handleException(final SQLException e) throws SQLException {
         fatalSqlExceptionThrown |= isFatalException(e);
+        if (fatalSqlExceptionThrown && getDelegate() != null) {
+            getDelegate().close();
+            this.close();
+        }
         super.handleException(e);
     }

What do you think about this approach?

Is it a completely dead-end or we can start working on it in this direction?

Do you agree that the reported and reproduced issue is a real one and nut just some kind of misconfiguration?

 

I am lost at this point and I need to move forward so I am asking for guidance here.

Issue Links

• is related to: OPENJPA-2919 Connection pool can be exhausted when connections are killed on the DB side
• is related to: OOZIE-3722 Workflow actions can stuck in RUNNING state when DB connections are killed on the DB side
• relates to: DBCP-513 Hundreads of threads in Wait state with below stack trace
• links to: GitHub Pull Request #348