[DERBY-4080] Deadlock between locks and latches in BTreeController.compareRowsForInsert()

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 10.4.2.0, 10.11.1.1
• Component/s: Store
• Labels: derby_triage10_5_2 derby_triage10_9
• Urgency: Normal
• Issue & fix info: Repro attached
• Assignee: Unassigned
• Reporter: Knut Anders Hatlen
• Created: 03/Mar/09 10:34
• Updated: 16/Apr/13 17:29

Description

It looks like BTreeController.compareRowsForInsert(), which is used to check for duplicates in a unique nullable index, can run into a deadlock which involves both locks and latches.

Here's what I think can happen:

comparePreviousRecord() (or compareNextRecord()) holds a latch on the index page where a new row is about to be inserted, and needs to check if there's a duplicate on one of the adjacent rows. Because the row is near a page boundary, this check moves to another index page, while still holding the latch on the original index page. Then compareRowsForInsert() is called, which tries to get an update lock on the existing row. If it has to wait for the update lock, the latch on the current page is released, but the latch on the original index page is kept. This means that the transaction is holding a latch while it is waiting for a lock, which means that it is blocking all access to that page until it has been granted the lock. If some other transaction that is holding a conflicting lock on the row later needs to latch the index page, those two transactions run into a deadlock and the one that's waiting for the lock will eventually time out (but it will not be reported as a deadlock).

If compareRowsForInsert() releases all latches when it needs to wait for a lock, the deadlock is prevented, and both of the transactions may be able to complete without timing out.