[DERBY-5552] Derby threads hanging when using ClientXADataSource and a deadlock or lock timeout occurs

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 10.1.3.1, 10.2.2.0, 10.3.3.0, 10.4.2.0, 10.5.3.0, 10.6.2.1, 10.7.1.1, 10.8.1.2, 10.8.2.2
• Fix Version/s: 10.5.3.1, 10.6.2.2, 10.7.1.4, 10.8.3.0, 10.9.1.0
• Component/s: Network Server
• Labels: derby_triage10_9
• Environment: Solaris 10, Glassfish V2.1.1,
• Urgency: Urgent
• Issue & fix info: Patch Available, Repro attached
• Bug behavior facts: Crash, Seen in production
• Assignee: Brett Wooldridge
• Reporter: Brett Bergquist
• Created: 22/Dec/11 15:54
• Updated: 15/Nov/13 08:15
• Resolved: 28/Feb/12 19:06

Description

The issue arrives when multiple XA transactions are done in parallel and there is either a lock timeout or a lock deadlock detected. When this happens the connection is leaked in the Glassfish connection pool and the client thread hangs in "org.apache.derby.client.netReply.fill(Reply.java:172)".

Shutting down the app server fails because the thread has a lock in "org.apache.derby.client.net.NetConnection40" and another task is calling "org.apache.derby.client.ClientPooledConnection.close(ClientPooledConnection.java:214)" which is waiting for the lock.

Killing the appsever using "kill" and then attempting to shutdown Derby network server causes the Network Server to hang. One of the threads hangs waiting for a lock at "org.apache.derby.impl.drda.NeworkServerControlImpl.removeFromSessionTable(NetworkServerControlImpl.java:1525)" and the "main" thread has this locked at "org.apache.derby.impl.drda.NetworkServerControlImpl.executeWork(NetworkServerControlImpl.java:2242)" and it itself is waiting for a lock which belongs to a thread that is stuck at "org.apache.derby.impl.services.locks.ActiveLock.waitForGrant(ActiveLock.java:118) which is in the TIMED_WAITING state.

Only by killing the Network Server using "kill" is possible at this point.

There are transactions left even though all clients have been removed.

Issue Links

• incorporates: DERBY-5633 Add tests for state transition and behavior with Lock Timeout in XA Transaction (DERBY-5552 testing task)
• relates to: DERBY-5553 System property for client tracing -Dderby.client.traceDirectory does not work with XADataSource