[DERBY-6510] Derby engine threads not making progress

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 10.9.1.0
• Component/s: Network Server
• Environment: Oracle Solaris 10/9, Oracle M5000 32 CPU, 128GB memory, 8GB allocated to Derby Network Server
• Urgency: Urgent
• Bug behavior facts: Performance, Seen in production
• Assignee: Unassigned
• Reporter: Brett Bergquist
• Created: 13/Mar/14 19:04
• Updated: 03/Sep/14 17:18

Description

We had an issue today in a production environment at a large customer site. Basically 5 database interactions became stuck and are not progressing. Part of the system dump performs a stack trace every few seconds for a period of a minute on the Glassfish application server and the Derby database engine (running in network server mode). Also, the dump captures the current transactions and the current lock table (ie. syscs_diag.transactions and syscs_diag.lock_table). We had to restart the system and in doing so, the Derby database engine would not shutdown and had to be killed.

The stack traces of the Derby engine show 5 threads that are basically making no progress in that at each sample, they are at the same point, waiting.

I will attach the stack traces as well as the state of the transactions and locks.

Interesting is that the "derby.jdbc.xaTransactionTimeout =1800" is set, yet the transactions did not timeout. The timeout is for 30 minutes but the transactions were in process for hours.