Deadlocks in ConcurrentVersioningWithTransactionsTest

• Key: JCR-962
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Bertrand Delacretaz
• Component/s: jackrabbit-core
• Fix Version/s: 1.3.3

Description

Patch follows for a ConcurrentVersioningWithTransactionsTest, based on the existing ConcurrentVersioningTest but using transactions around the versioning operations.

On my macbook, running the test with CONCURRENCY = 100 and NUM_OPERATIONS = 100 causes a deadlock after a few seconds, thread dumps follow.

Note that I had to ignore StaleItemStateException (which is probably justified, due to not locking stuff IIUC) to let the threads run long enough to show the problem.

Running the test a few times showed the same locking pattern several times: some threads are locked at line 87 (session.save(), no transaction) while others are at line 93 (transaction.commit()), in testConcurrentCheckinInTransaction():

    80 public void testConcurrentCheckinInTransaction() throws RepositoryException {
    81 runTask(new Task() {
    82 public void execute(Session session, Node test) throws RepositoryException {
    83 int i = 0;
    84 try {
    85 Node n = test.addNode("test");
    86 n.addMixin(mixVersionable);
    87 session.save();
    88 for (i = 0; i < NUM_OPERATIONS / CONCURRENCY; i++) {
    89 final UserTransaction utx = new UserTransactionImpl(test.getSession());
    90 utx.begin();
    91 n.checkout();
    92 n.checkin();
    93 utx.commit();
    94 }
    95 n.checkout();
    96 } catch (Exception e) {
    97 final String threadName = Thread.currentThread().getName();
    98 final Throwable deepCause = getLevel2Cause(e);
    99 if(deepCause!=null && deepCause instanceof StaleItemStateException) {
   100 // ignore
   101 } else {
   102 throw new RepositoryException(threadName + ", i=" + i + ":" + e.getClass().getName(), e);
   103 }
   104 }
   105 }
   106 }, CONCURRENCY);
   107 }

Comments

[Bertrand Delacretaz]

ConcurrentVersioningWithTransactionsTest patch, must be enabled in org.apache.jackrabbit.core.TestAll

[Bertrand Delacretaz]

Deadlocked threads dump with 10 concurrent tasks

[Bertrand Delacretaz]

Deadlocked threads dump with 100 concurrent tasks

[Marcel Reutegger]

Attached the stacktraces of the two relevant threads from 10-threads-dump.txt, which are in a deadlocked state.

[Marcel Reutegger]

Added Bertrands' test case and created a new test suite, which contains all long running tests related to multi-threading: MultiThreadingTests. The suite will not run with the regression test while building jackrabbit-core but can easily be started in an IDE.

[Marcel Reutegger]

This patch replaces the internal XAResources on the XAWorkspace with two similar ones on the XAVersionManager. The idea is to lock the version manager early in the prepare phase and release the lock at the end of the transaction. When changes on the workspace are committed the version manager is already locked and stays locked until the version changes are committed.

If there are no version items in the transaction the write lock on the version manager is not acquired to allow concurrency between multiple sessions operating on different workspaces.

With this patch the sequence for locking workspace and version store is:
1) version store
2) workspace

The rest of the jackrabbit core should be reviewed and changed accordingly.

[Marcel Reutegger]

Assuming lazy consensus I committed the patch in revision: 551877

[Jukka Zitting]

Merged to the 1.3 branch in revision 581177.