[JCR-4674] Deadlock at ItemManager.getItem

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.20.0
• Component/s: jackrabbit-core
• Assignee: Unassigned
• Reporter: Hannes Weichel
• Created: 23/Feb/21 16:20
• Updated: 03/Jan/23 12:23
• Resolved: 29/Apr/21 13:11

Description

We run a Jackrabbit in a Tomcat (8.5) system with many parallel user ( > 30000 parallel user ). Jackrabbit runs mostly fast and stable. But at a certain point of load the threads are blocking each other. Most threads are blocked at the following position:

(~28000 threads all blocking):
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:621)

(~2000 threads all blocking):
org.apache.jackrabbit.core.state.DefaultISMLocking.acquireReadLock(DefaultISMLocking.java:114)

Even when we block further requests, many threads keep in blocked state. We must hard restart tomcat and throttle user requests to get the system (very slow) back to a working state. 

What we tried so far to prevent this behavior (with no success):

• configured the bundleCacheSize (see attached repository.xml )
• configured derbyStoragePageCacheSize
• used a JCR-Session Pool
• adding <ISMLocking class="org.apache.jackrabbit.core.state.FineGrainedISMLocking"/> in repository.xml 
• give the system more Hardware (20 Cores, 128 GB RAM, faster Storage) - but hardware is not the problem.
• fine tuned tomcat for more und faster threads

 

here some example StackTraces:

 

http-nio-8080-exec-11367
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:621)
org.apache.jackrabbit.core.security.authorization.acl.CompiledPermissionsImpl.canRead(CompiledPermissionsImpl.java:254)
org.apache.jackrabbit.core.security.DefaultAccessManager.canRead(DefaultAccessManager.java:265)
org.apache.jackrabbit.core.ItemManager.canRead(ItemManager.java:437)
org.apache.jackrabbit.core.ItemManager.createItemData(ItemManager.java:847)
org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:391)
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:328)
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:621)
org.apache.jackrabbit.core.NodeImpl$11.perform(NodeImpl.java:2258)
org.apache.jackrabbit.core.NodeImpl$11.perform(NodeImpl.java:2252)
org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91)
org.apache.jackrabbit.core.NodeImpl.getProperty(NodeImpl.java:2252)
...

 

http-nio-8080-exec-11352
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:621)
org.apache.jackrabbit.core.security.authorization.acl.CompiledPermissionsImpl.canRead(CompiledPermissionsImpl.java:254)
org.apache.jackrabbit.core.security.DefaultAccessManager.canRead(DefaultAccessManager.java:265)
org.apache.jackrabbit.core.ItemManager.canRead(ItemManager.java:437)
org.apache.jackrabbit.core.ItemManager.createItemData(ItemManager.java:847)
org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:391)
org.apache.jackrabbit.core.ItemManager.itemExists(ItemManager.java:304)
org.apache.jackrabbit.core.ItemManager.itemExists(ItemManager.java:522)
org.apache.jackrabbit.core.NodeImpl.hasProperty(NodeImpl.java:2460)
...

http-nio-8080-exec-11382
org.apache.jackrabbit.core.ItemManager.getItem(ItemManager.java:621)
org.apache.jackrabbit.core.security.authorization.acl.CompiledPermissionsImpl.canRead(CompiledPermissionsImpl.java:254)
org.apache.jackrabbit.core.security.DefaultAccessManager.canRead(DefaultAccessManager.java:265)
org.apache.jackrabbit.core.ItemManager.canRead(ItemManager.java:437)
org.apache.jackrabbit.core.ItemManager.createItemData(ItemManager.java:847)
org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:391)
org.apache.jackrabbit.core.ItemManager.getNode(ItemManager.java:668)
org.apache.jackrabbit.core.ItemManager.getNode(ItemManager.java:646)
org.apache.jackrabbit.core.LazyItemIterator.prefetchNext(LazyItemIterator.java:120)
org.apache.jackrabbit.core.LazyItemIterator.next(LazyItemIterator.java:257)
org.apache.jackrabbit.core.LazyItemIterator.nextNode(LazyItemIterator.java:166)
...

http-nio-8080-exec-11357
org.apache.jackrabbit.core.state.DefaultISMLocking.acquireReadLock(DefaultISMLocking.java:114)
org.apache.jackrabbit.core.state.SharedItemStateManager.acquireReadLock(SharedItemStateManager.java:1893)
org.apache.jackrabbit.core.state.SharedItemStateManager.getItemState(SharedItemStateManager.java:268)
org.apache.jackrabbit.core.state.LocalItemStateManager.getPropertyState(LocalItemStateManager.java:135)
org.apache.jackrabbit.core.state.LocalItemStateManager.getItemState(LocalItemStateManager.java:177)
org.apache.jackrabbit.core.state.XAItemStateManager.getItemState(XAItemStateManager.java:260)
org.apache.jackrabbit.core.state.SessionItemStateManager.getItemState(SessionItemStateManager.java:161)
org.apache.jackrabbit.core.ItemManager.getItemData(ItemManager.java:382)
org.apache.jackrabbit.core.ItemManager.itemExists(ItemManager.java:304)
org.apache.jackrabbit.core.ItemManager.itemExists(ItemManager.java:522)
org.apache.jackrabbit.core.NodeImpl.hasProperty(NodeImpl.java:2460)