Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Not A Problem
Affects Version/s: 2.11.0
Fix Version/s: None
Labels:
- deadlock
- robustness

Language:
- java

Description

Apache Oozie 5.2.1 uses OpenJPA 2.4.2 and commons-dbcp 1.4 and commons-pool 1.5.4. These are ancient versions, I know.

Description

The issue is that when due to some network issues or "maintenance work" on the DB side (especially PostgreSQL) which causes the DB connection to be closed, it results exhausted Pool on the client side. Many threads are waiting at this point:

"pool-2-thread-4" #20 prio=5 os_prio=31 tid=0x00007faf7903b800 nid=0x8603 waiting on condition [0x000000030f3e7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000066aca8e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:1324)

According to my observation this is because the JDBC driver does not get closed on the client side, nor the abstract DBCP connection org.apache.commons.dbcp2.PoolableConnection .

Repro

(Un)Fortunately I can reproduce the issue using the latest and greatest commons-dbcp 2.11.0 and commons-pool 2.12.0 along with OpenJPA 3.2.2.

I've just created a Java application to reproduce the issue: https://github.com/dionusos/pool_exhausted_repro . See README.md for detailed repro steps.

Kind of solution?

To be honest I am not really familiar with DBCP but with this change I managed to make my application more robust:

diff --git a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
index 440cb756..678550bf 100644
--- a/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
+++ b/src/main/java/org/apache/commons/dbcp2/PoolableConnection.java
@@ -214,6 +214,10 @@ public class PoolableConnection extends DelegatingConnection<Connection> impleme
     @Override
     protected void handleException(final SQLException e) throws SQLException {
         fatalSqlExceptionThrown |= isFatalException(e);
+        if (fatalSqlExceptionThrown && getDelegate() != null) {
+            getDelegate().close();
+            this.close();
+        }
         super.handleException(e);
     }

What do you think about this approach?

Is it a completely dead-end or we can start working on it in this direction?

Do you agree that the reported and reproduced issue is a real one and nut just some kind of misconfiguration?

I am lost at this point and I need to move forward so I am asking for guidance here.