Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.6.1, 2.0
    • Fix Version/s: 1.6.2, 2.1
    • Component/s: jackrabbit-core
    • Labels:
    • Environment:
      Weblogic 9.2

      Description

      In one of our client deployments on WebLogic 9.2 we observed JackRabbit sessions going stale in a load test. This was observed against release 1.6.1 (to which we migrated due to concurrency related issues JCR-2081 and JCR-2237). Same effect with 2.0.0.

      I could finally reproduce this issue locally. And it seems to boil down to WLS invoking the sequence of <prepare> ... <release> ... <commit> on one XA session from multiple threads, as it seems breaking assumptions of the thread-bound java.util.concurrent-RWLock based DefaultISMLocking class.
      Effectively the setActiveXid(..) method on DefaultISMLocking$RWLock fails as the old active XID was not yet cleared. With the result of more and more sessions deadlocking in below's invocation stack.

      "[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=1 tid=0x33fc3ec0 nid=0x2324 in Object.wait() [0x2156a000..0x2156beb0] at java.lang.Object.wait(Native Method) - waiting on <0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock) at java.lang.Object.wait(Object.java:474) at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(Unknown Source) - locked <0x68a54698> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock) at org.apache.jackrabbit.core.state.DefaultISMLocking$1.<init>(DefaultISMLocking.java:64) at org.apache.jackrabbit.core.state.DefaultISMLocking.acquireWriteLock(DefaultISMLocking.java:61) at org.apache.jackrabbit.core.version.AbstractVersionManager.acquireWriteLock(AbstractVersionManager.java:146) at org.apache.jackrabbit.core.version.XAVersionManager$1.prepare(XAVersionManager.java:562) at org.apache.jackrabbit.core.TransactionContext.prepare(TransactionContext.java:154) - locked <0x6dc2ad88> (a org.apache.jackrabbit.core.TransactionContext) at org.apache.jackrabbit.core.XASessionImpl.prepare(XASessionImpl.java:331) at org.apache.jackrabbit.jca.TransactionBoundXAResource.prepare(TransactionBoundXAResource.java:68) at weblogic.connector.security.layer.AdapterLayer.prepare(AdapterLayer.java:397) at weblogic.connector.transaction.outbound.XAWrapper.prepare(XAWrapper.java:297) at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:1276) at weblogic.transaction.internal.XAServerResourceInfo.prepare(XAServerResourceInfo.java:499) at weblogic.transaction.internal.ServerSCInfo$1.execute(ServerSCInfo.java:335) at weblogic.kernel.Kernel.executeIfIdle(Kernel.java:243) at weblogic.transaction.internal.ServerSCInfo.startPrepare(ServerSCInfo.java:326) at weblogic.transaction.internal.ServerTransactionImpl.localPrepare(ServerTransactionImpl.java:2516) at weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2211) at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:266) at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227) at weblogic.transaction.internal.TransactionManagerImpl.commit(TransactionManagerImpl.java:283) at org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1028) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:709) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:678)
      

        Attachments

        1. LockManager.patch
          4 kB
          Claus Köll
        2. Xid_v4.patch
          12 kB
          Claus Köll
        3. Xid_v3.patch
          5 kB
          Claus Köll
        4. Xid_v2 + SynchronizedRef for markedXid.jpg
          63 kB
          Robert Sauer
        5. Xid_v2 deadlock.jpg
          62 kB
          Robert Sauer
        6. Xid_v2.patch
          5 kB
          Claus Köll
        7. Xid.patch
          3 kB
          Claus Köll
        8. ConcurrentReaders.java
          5 kB
          Robert Sauer
        9. jr-core-xid-aware-ism-locking1.patch
          16 kB
          Robert Sauer

          Activity

            People

            • Assignee:
              c_koell Claus Köll
              Reporter:
              rsauer Robert Sauer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: