[JCR-3803] Local cache contention lead to performance degradation

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8
• Fix Version/s: 2.9.1
• Component/s: jackrabbit-data
• Assignee: Amit Jain
• Reporter: Shashank Gupta
• Created: 14/Aug/14 04:33
• Updated: 06/Feb/15 13:54
• Resolved: 02/Sep/14 04:43

Description

stack trace of few threads. (full dump attached) one thread holding monitor <0x000000040298beb0> while other BLOCKED on it.

"115.248.212.49 [1407765540589] GET /libs/dam/gui/content/commons/sidepanels/search/items/searchpanel/result/content/items/searchpages.html HTTP/1.1" prio=10 tid=0x00007f27e0001000 nid=0x2503 runnable [0x00007f2458468000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
	at sun.security.ssl.InputRecord.read(InputRecord.java:509)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	- locked <0x0000000705de7ea8> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	- locked <0x0000000705df1008> (a sun.security.ssl.AppInputStream)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:204)
	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:182)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.apache.jackrabbit.core.data.LocalCache.store(LocalCache.java:145)
	- locked <0x000000040298beb0> (a org.apache.jackrabbit.core.data.LocalCache)
	at org.apache.jackrabbit.core.data.CachingDataStore.getLength(CachingDataStore.java:562)
	at org.apache.jackrabbit.core.data.CachingDataRecord.getLength(CachingDataRecord.java:54)
	at org.apache.jackrabbit.core.value.BLOBInDataStore.getSize(BLOBInDataStore.java:96)

"115.248.212.49 [1407766318844] GET /assets.html/content/dam/archive-2001-2014/Content%20From%20HDD HTTP/1.1" prio=10 tid=0x00007f27e800a800 nid=0x28bc waiting for monitor entry [0x00007f2458661000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.jackrabbit.core.data.LocalCache.getFileIfStored(LocalCache.java:244)
	- waiting to lock <0x000000040298beb0> (a org.apache.jackrabbit.core.data.LocalCache)
	at org.apache.jackrabbit.core.data.CachingDataStore.getRecord(CachingDataStore.java:394)
	- locked <0x0000000402f6ade0> (a org.apache.jackrabbit.aws.ext.ds.S3DataStore)
	at org.apache.jackrabbit.core.value.BLOBInDataStore.getDataRecord(BLOBInDataStore.java:151)
	at org.apache.jackrabbit.core.value.BLOBInDataStore.getSize(BLOBInDataStore.java:96)
	at org.apache.jackrabbit.core.value.InternalValue.getLength(InternalValue.java:654)
	at org.apache.jackrabbit.core.PropertyImpl.getLength(PropertyImpl.java:237)
	at org.apache.jackrabbit.core.PropertyImpl.getLength(PropertyImpl.java:835)
	at org.apache.sling.jcr.resource.internal.helper.jcr.JcrNodeResource.setMetaData(JcrNodeResource.java:307)

"115.248.212.49 [1407766115000] GET /libs/dam/gui/content/commons/sidepanels/search/items/searchpanel/result.html HTTP/1.1" prio=10 tid=0x00007f27e4028000 nid=0x2712 waiting for monitor entry [0x00007f2457490000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.jackrabbit.core.data.LocalCache.store(LocalCache.java:138)
	- waiting to lock <0x000000040298beb0> (a org.apache.jackrabbit.core.data.LocalCache)
	at org.apache.jackrabbit.core.data.CachingDataStore.getLength(CachingDataStore.java:562)
	at org.apache.jackrabbit.core.data.CachingDataRecord.getLength(CachingDataRecord.java:54)
	at org.apache.jackrabbit.core.value.BLOBInDataStore.getSize(BLOBInDataStore.java:96)
	at org.apache.jackrabbit.core.value.InternalValue.getLength(InternalValue.java:654)
	at org.apache.jackrabbit.core.PropertyImpl.getLength(PropertyImpl.java:237)
	at org.apache.jackrabbit.core.PropertyImpl.getLength(PropertyImpl.java:835)
	at org.apache.sling.jcr.resource.internal.helper.jcr.JcrNodeResource.setMetaData(JcrNodeResource.java:307)