[GEODE-2878] If an exception occurs after retrieving an XAConnection from the ConnectionProvider but before returning it to the application, the GemFireTransactionDataSource doesn't return it to the pool

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.10.0
• Component/s: transactions
• Labels: needs-review pull-request-available storage_3
• Assignee: Mario Ivanac
• Reporter: Barrett Oglesby
• Created: 04/May/17 23:54
• Updated: 26/Sep/19 18:05
• Resolved: 18/Jun/19 20:24

Description

In my test, I have 5 threads inserting rows into a derby database.

At first, as connections are being used and returned, the activeConnections is updated correctly:

Thread-16: AbstractPoolCache.getPooledConnectionFromPool activeConnections=1
Thread-15: AbstractPoolCache.getPooledConnectionFromPool activeConnections=2
Thread-17: AbstractPoolCache.getPooledConnectionFromPool activeConnections=3
Thread-14: AbstractPoolCache.getPooledConnectionFromPool activeConnections=4
Thread-18: AbstractPoolCache.getPooledConnectionFromPool activeConnections=5
Thread-16: AbstractPoolCache.returnPooledConnectionToPool activeConnections=4
Thread-14: AbstractPoolCache.returnPooledConnectionToPool activeConnections=3
Thread-18: AbstractPoolCache.returnPooledConnectionToPool activeConnections=2
Thread-17: AbstractPoolCache.returnPooledConnectionToPool activeConnections=1
Thread-15: AbstractPoolCache.returnPooledConnectionToPool activeConnections=0

But, then if an exception occurs after retrieving the XAConnection, it is not return to the ConnectionProvider.

In my test, the exception occurs in GemFireTransactionDataSource.registerTranxConnection:

java.lang.Exception: GemFireTransactionDataSource-registerTranxConnection(). Exception in registering the XAResource with the Transaction.Exception occurred= javax.transaction.SystemException: GlobalTransaction::enlistResource::error while enlisting XAResource org.apache.derby.client.am.XaException: XAER_RMFAIL : An error occurred during a deferred connect reset and the connection has been terminated.
	at org.apache.geode.internal.datasource.GemFireTransactionDataSource.registerTranxConnection(GemFireTransactionDataSource.java:218)
	at org.apache.geode.internal.datasource.GemFireTransactionDataSource.getConnection(GemFireTransactionDataSource.java:127)
	at TestServer.saveToDB(TestServer.java:177)
	at TestServer.save(TestServer.java:154)
	at TestServer.loadEntriesIntoDerby(TestServer.java:127)
	at TestServer$1.run(TestServer.java:112)
	at java.lang.Thread.run(Thread.java:745)

This is after the XAConnection has been retrieved from the ConnectionProvider and the activeConnections incremented, but before it has been returned to the application. Neither the registerTranxConnection method nor its caller (getConnection) does anything other than to throw the exception. The XAConnection is not returned to the pool nor is the activeConnections decremented.

Finally, if enough of these exceptions occur, the test stops because all 30 (default max) connections are in use. They aren't really in use, its just that the activeConnections counter hasn't been properly maintained.

Thread-14: AbstractPoolCache.returnPooledConnectionToPool activeConnections=28
Thread-15: AbstractPoolCache.getPooledConnectionFromPool activeConnections=29
Thread-14: AbstractPoolCache.getPooledConnectionFromPool activeConnections=30
Thread-16: AbstractPoolCache.returnPooledConnectionToPool activeConnections=29
Thread-18: AbstractPoolCache.returnPooledConnectionToPool activeConnections=28
Thread-15: AbstractPoolCache.getPooledConnectionFromPool activeConnections=29
Thread-17: AbstractPoolCache.getPooledConnectionFromPool activeConnections=30
Thread-14: AbstractPoolCache.returnPooledConnectionToPool activeConnections=29
Thread-18: AbstractPoolCache.getPooledConnectionFromPool activeConnections=30
Thread-17: AbstractPoolCache.returnPooledConnectionToPool activeConnections=29
Thread-14: AbstractPoolCache.getPooledConnectionFromPool activeConnections=30

It doesn't really matter what the exception is. If one occurs after retrieving the XAConnection, it needs to be returned to the ConnectionProvider or at the very least, the activeConnections must be decremented.

Issue Links

• links to: GitHub Pull Request #3538