[JCR-3888] Possible dead lock in SharedItemStateManager in clustered environment

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.1
• Fix Version/s: 2.10.2, 2.11.0
• Assignee: Unico Hommes
• Reporter: Unico Hommes
• Created: 23/Jun/15 08:46
• Updated: 17/Aug/15 07:57
• Resolved: 05/Aug/15 10:49

Description

When using clustering SharedItemStateManager#Update#begin sets a lock on the database (line 573) and subsequently a write lock is set (line 579):

            // let listener know about change
            try {
                eventChannel.updateCreated(this); // line 573
            } catch (ClusterException e) {
                throw new ItemStateException(e.getMessage(), e);
            }

            try {
                writeLock = acquireWriteLock(local); // line 579
            } finally {
                if (writeLock == null) {
                    eventChannel.updateCancelled(this); // line 582
                }
            }

In SharedItemStateManager#Update#end the update is committed and the write lock released:

                eventChannel.updateCommitted(this, path); // line 849
                setAttribute(ATTRIBUTE_UPDATE_SIZE, null);

                if (writeLock != null) {
                    // exception occurred before downgrading lock
                    writeLock.release();
                    writeLock = null;
                } else if (readLock != null) {
                    try {
                        if (succeeded) {
                            /* dispatch the events */
                            events.dispatch();
                        }
                    } finally {
                        readLock.release(); // 863
                    }
                }

However, the implementation of eventChannel#updateCommitted in ClusterNode#WorkspaceUpdateChannel#updateCommitted has the following code:

                long journalUpdateSize = record.update(); // line 700

                long recordRevision = record.getRevision();
                setRevision(recordRevision); // line 703

The database lock is freed during the call to update the record on line 700.
If a contending thread now sets a database lock at SharedItemStateManager:573 before this thread can set the revision in the database at ClusterNode:703 we have a dead lock. This is because this thread holds the write lock in SharedItemStateManager causing the contending thread to wait at SharedItemStateManager:579 while the contending thread holds the database lock causing this thread to wait at ClusterNode:703