[DERBY-4279] Statement cache deadlock

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 10.0.2.1, 10.1.3.1, 10.2.2.0, 10.3.3.0, 10.4.2.0, 10.5.1.1, 10.8.1.2
• Fix Version/s: 10.8.3.0, 10.9.2.2, 10.10.1.1
• Component/s: SQL
• Labels: derby_triage10_5_2
• Environment: Windows Vista, OS X 10.5+
• Urgency: Normal
• Issue & fix info: Repro attached
• Bug behavior facts: Seen in production
• Assignee: Brett Wooldridge
• Reporter: Jeff Stuckman
• Created: 19/Jun/09 23:29
• Updated: 03/Sep/14 08:31
• Resolved: 12/Sep/12 17:09

Description

Due to a design flaw in the statement cache, a deadlock can occur if a prepared statement becomes out-of-date.

I will illustrate this with the following example:

The application is using the embedded Derby driver. The application has two threads, and each thread uses its own connection.

There is a table named MYTABLE with column MYCOLUMN.

1. A thread prepares and executes the query SELECT MYCOLUMN FROM MYTABLE. The prepared statement is stored in the statement cache (see org.apache.derby.impl.sql.GenericStatement for this logic)
2. After some time, the prepared statement becomes invalid or out-of-date for some reason (see org.apache.derby.impl.sql.GenericPreparedStatement)
3. Thread 1 begins a transaction and executes LOCK TABLE MYTABLE IN EXCLUSIVE MODE
4. Thread 2 begins a transaction and executes SELECT MYCOLUMN FROM MYTABLE. The statement is in the statement cache but it is out-of-date. The thread begins to recompile the statement. To compile the statement, the thread needs a shared lock on MYTABLE. Thread 1 already has an exclusive lock on MYTABLE. Thread 2 waits.
5. Thread 1 executes SELECT MYCOLUMN FROM MYTABLE. The statement is in the statement cache but it is being compiled. Thread 1 waits on the statement's monitor.
6. We have a deadlock. Derby eventually detects a lock timeout, but the error message is not descriptive. The stacks at the time of the deadlock are:

This deadlock is unique because it can still occur in a properly designed database. You are only safe if all of your transactions are very simple and cannot be interleaved in a sequence that causes the deadlock, or if your particular statements do not require a table lock to compile. (For the sake of simplicity, I used LOCK TABLE in my example, but any UPDATE statement would fit.)

Issue Links

• incorporates: DERBY-4721 NullPointerException on StressMultiTest in language layer
• is duplicated by: DERBY-5703 Threads are stuck in ActiveLock.waitForGrant