Uploaded image for project: 'Jackrabbit Content Repository'
  1. Jackrabbit Content Repository
  2. JCR-2822

Deadlock on cluster during making changes on content repository

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.6.4, 2.1.2
    • None
    • clustering, jackrabbit-core
    • None
    • - JEE Web application with parallel web sessions
      - Several clusters of JEE server
      - Oracle database

    Description

      Deadlock is possible on cluster during making changes on content repository.
      The steps for deadlock are following:

      Thread 1 - making some changes on content repository. Code:
      public static void move(final Node node, final Node destNode, final Session session,
      final Boolean versionable) throws ItemNotFoundException, RepositoryException {
      final String path = node.getPath();
      final String name = node.getName();
      final String destPath = destNode.getPath() + FileSystem.SEPARATOR_CHAR + name;
      session.move(path, destPath);
      session.save();

      Thread 2 - tries to commit transaction and executes org.apache.jackrabbit.core.TransactionContext.prepare()

      What's happened:

      Thread 1:

      • Call session.move
      • ..., calls CachingHierarchyManager.nodeRemoved, it locks cacheMonitor
      • thread 2 in this time does sync and executes SharedItemStateManager.externalUpdate, which occurs write lock
      • thread 1 continues working and tries to occur read lock on SharedItemStateManager.hasItemState, but write lock is already on thread 2, so thread 1 starts wait

      Thread 2:

      • Calls TransactionBoundXAResource.prepare
      • comes to ClusterNode.consume
      • Executes SharedItemStateManager.externalUpdate, where occurs the write lock
      • And finally comes to CachingHierarchyManager.nodeModified, but can't proceed because of cacheMonitor is already locked by Thread1

      Thrace from Thread1:
      null at java.lang.Object.wait(J)V(Native Method)

      • waiting on <0xfffffffe7e84c348> (a EDU.oswego.cs.dl.util.concurrent.Latch)
        at java.lang.Object.wait()V(Object.java:474)
        at EDU.oswego.cs.dl.util.concurrent.Latch.acquire()V(Unknown Source)
      • locked <0xfffffffe7e84c348> (a EDU.oswego.cs.dl.util.concurrent.Latch)
        at org.apache.jackrabbit.core.state.FineGrainedISMLocking.acquireReadLock(Lorg/apache/jackrabbit/core/ItemId;)Lorg/apache/jackrabbit/core/state/ISMLocking$ReadLock;(FineGrainedISMLocking.java:112)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(Lorg/apache/jackrabbit/core/ItemId;)Lorg/apache/jackrabbit/core/state/ISMLocking$ReadLock;(SharedItemStateManager.java:1432)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(SharedItemStateManager.java:286)
        at org.apache.jackrabbit.core.state.XAItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(XAItemStateManager.java:302)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(SessionItemStateManager.java:218)
        at org.apache.jackrabbit.core.HierarchyManagerImpl.hasItemState(Lorg/apache/jackrabbit/core/ItemId;)Z(HierarchyManagerImpl.java:164)
        at org.apache.jackrabbit.core.CachingHierarchyManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Path;Lorg/apache/jackrabbit/core/NodeId;)V(CachingHierarchyManager.java:725)
        at org.apache.jackrabbit.core.CachingHierarchyManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(CachingHierarchyManager.java:455)
      • locked <0xfffffffe63626110> (a java.lang.Object)
        at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyNodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(StateChangeDispatcher.java:207)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.nodeRemoved(Lorg/apache/jackrabbit/core/state/NodeState;Lorg/apache/jackrabbit/spi/Name;ILorg/apache/jackrabbit/core/NodeId;)V(SessionItemStateManager.java:983)
        at org.apache.jackrabbit.core.state.NodeState.notifyNodeRemoved(Lorg/apache/jackrabbit/core/state/ChildNodeEntry;)V(NodeState.java:924)
        at org.apache.jackrabbit.core.state.NodeState.removeChildNodeEntry(Lorg/apache/jackrabbit/spi/Name;I)Z(NodeState.java:403)
        at org.apache.jackrabbit.core.SessionImpl.move(Ljava/lang/String;Ljava/lang/String;)V(SessionImpl.java:1118)
        at org.apache.jackrabbit.jca.JCASessionHandle.move(Ljava/lang/String;Ljava/lang/String;)V(JCASessionHandle.java:170)
        at mypackage.ContentRepositoryUtil.move(Ljavax/jcr/Node;Ljavax/jcr/Node;Ljavax/jcr/Session;)V(ContentRepositoryUtil.java:163)

      Thread 2:
      null at org.apache.jackrabbit.core.CachingHierarchyManager.nodeModified(Lorg/apache/jackrabbit/core/state/NodeState;)V(CachingHierarchyManager.java:306)

      • waiting to lock <0xfffffffe63626110> (a java.lang.Object)
        at org.apache.jackrabbit.core.CachingHierarchyManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(CachingHierarchyManager.java:293)
        at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
        at org.apache.jackrabbit.core.state.SessionItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(SessionItemStateManager.java:889)
        at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
        at org.apache.jackrabbit.core.state.LocalItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(LocalItemStateManager.java:452)
        at org.apache.jackrabbit.core.state.XAItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(XAItemStateManager.java:602)
        at org.apache.jackrabbit.core.state.StateChangeDispatcher.notifyStateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(StateChangeDispatcher.java:111)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.stateModified(Lorg/apache/jackrabbit/core/state/ItemState;)V(SharedItemStateManager.java:400)
        at org.apache.jackrabbit.core.state.ItemState.notifyStateUpdated()V(ItemState.java:244)
        at org.apache.jackrabbit.core.state.ChangeLog.persisted()V(ChangeLog.java:297)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.doExternalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;)V(SharedItemStateManager.java:1209)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.externalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Lorg/apache/jackrabbit/core/observation/EventStateCollection;)V(SharedItemStateManager.java:1132)
        at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.externalUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Ljava/util/List;JLjava/lang/String;)V(RepositoryImpl.java:2193)
        at org.apache.jackrabbit.core.cluster.ClusterNode.process(Lorg/apache/jackrabbit/core/cluster/ChangeLogRecord;)V(ClusterNode.java:869)
        at org.apache.jackrabbit.core.cluster.ChangeLogRecord.process(Lorg/apache/jackrabbit/core/cluster/ClusterRecordProcessor;)V(ChangeLogRecord.java:507)
        at org.apache.jackrabbit.core.cluster.ClusterNode.consume(Lorg/apache/jackrabbit/core/journal/Record;)V(ClusterNode.java:815)
        at org.apache.jackrabbit.core.journal.AbstractJournal.doSync(J)V(AbstractJournal.java:214)
        at org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync()V(AbstractJournal.java:258)
        at org.apache.jackrabbit.core.journal.DefaultRecordProducer.append()Lorg/apache/jackrabbit/core/journal/Record;(DefaultRecordProducer.java:51)
        at org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceUpdateChannel.updateCreated(Lorg/apache/jackrabbit/core/cluster/Update;)V(ClusterNode.java:602)
        at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin()V(SharedItemStateManager.java:554)
        at org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate(Lorg/apache/jackrabbit/core/state/ChangeLog;Lorg/apache/jackrabbit/core/observation/EventStateCollectionFactory;Lorg/apache/jackrabbit/core/virtual/VirtualItemStateProvider;)Lorg/apache/jackrabbit/core/state/SharedItemStateManager$Update;(SharedItemStateManager.java:1085)
        at org.apache.jackrabbit.core.state.XAItemStateManager.prepare(Lorg/apache/jackrabbit/core/TransactionContext;)V(XAItemStateManager.java:163)
        at org.apache.jackrabbit.core.TransactionContext.prepare()V(TransactionContext.java:153)
      • locked <0xfffffffe7e4b17a8> (a org.apache.jackrabbit.core.TransactionContext)
        at org.apache.jackrabbit.core.XASessionImpl.prepare(Ljavax/transaction/xa/Xid;)I(XASessionImpl.java:331)
        at org.apache.jackrabbit.jca.TransactionBoundXAResource.prepare(Ljavax/transaction/xa/Xid;)I(TransactionBoundXAResource.java:68)
        at mypackage.XAResourceWrapper.prepare()I(XAResourceWrapper.java:61)

      repository.xml:
      <?xml version="1.0"?>
      <Repository>
      <FileSystem class="mypackage.OracleFileSystem">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@loclhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="schema" value="oracle" />
      <param name="schemaObjectPrefix" value="PREF_" />
      </FileSystem>
      <Security appName="Jackrabbit">
      <AccessManager
      class="org.apache.jackrabbit.core.security.simple.SimpleAccessManager">
      </AccessManager>
      <LoginModule class="org.apache.jackrabbit.core.security.SimpleLoginModule">
      <param name="anonymousId" value="anonymous" />
      </LoginModule>
      </Security>
      <Workspaces rootPath="${rep.home}/workspaces"
      defaultWorkspace="default" />
      <Workspace name="${wsp.name}">
      <FileSystem class="mypackage.OracleFileSystem">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@loclhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="schema" value="oracle" />
      <param name="schemaObjectPrefix" value="PREF_" />
      </FileSystem>
      <PersistenceManager
      class="mypackage.OracleBundlePersistenceManager">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@loclhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="schema" value="oracle" />
      <param name="schemaObjectPrefix" value="P_${wsp.name}_" />
      </PersistenceManager>
      <ISMLocking class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
      </Workspace>

      <Versioning rootPath="${rep.home}/version">
      <FileSystem class="mypackage.OracleFileSystem">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@localhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="schema" value="oracle" />
      <param name="schemaObjectPrefix" value="P_V_" />
      </FileSystem>
      <PersistenceManager
      class="mypackage.OracleBundlePersistenceManager">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@localhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="schema" value="oracle" />
      <param name="schemaObjectPrefix" value="P_V_" />
      </PersistenceManager>

      <ISMLocking class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/>
      </Versioning>
      <DataStore class="org.apache.jackrabbit.core.data.db.DbDataStore">
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url"
      value="jdbc:oracle:thin:@localhost" />
      <param name="user" value="" />
      <param name="password" value="" />
      <param name="databaseType" value="oracle" />
      <param name="maxConnections" value="50" />
      <param name="tablePrefix" value="SO_CM_"/>
      </DataStore>

      <Cluster id="node1" syncDelay="2000">
      <Journal class="org.apache.jackrabbit.core.journal.OracleDatabaseJournal">
      <param name="revision" value="${rep.home}/revision.log" />
      <param name="driver" value="oracle.jdbc.driver.OracleDriver" />
      <param name="url" value="jdbc:oracle:thin:@localhost" />
      <param name="user" value="" />
      <param name="password" value=""/>
      <param name="schemaObjectPrefix" value="PREF_" />
      </Journal>
      </Cluster>
      </Repository>

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. JCR-3457 Possible deadlock during external update

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              alshpak aliaksandr shpak
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: