[OAK-2761] Persistent cache: add data in a different thread

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.18, 1.3.16, 1.4
• Component/s: cache, core, mongomk
• Labels: resilience
• Epic Link: Persistent Cache improvements
• Assignee: Thomas Mueller
• Reporter: Thomas Mueller
• Created: 14/Apr/15 06:59
• Updated: 08/Oct/19 15:21
• Resolved: 16/Feb/16 10:39

Description

The persistent cache usually stores data in a background thread, but sometimes (if a lot of data is added quickly) the foreground thread is blocked.

Even worse, switching the cache file can happen in a foreground thread, with the following stack trace.

"127.0.0.1 [1428931262206] POST /bin/replicate.json HTTP/1.1" prio=5 tid=0x00007fe5df819800 nid=0x9907 runnable [0x0000000113fc4000]
   java.lang.Thread.State: RUNNABLE
        ...
	at org.h2.mvstore.MVStoreTool.compact(MVStoreTool.java:404)
	at org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1.closeStore(PersistentCache.java:213)
	- locked <0x0000000782483050> (a org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache$1)
	at org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache.switchGenerationIfNeeded(PersistentCache.java:350)
	- locked <0x0000000782455710> (a org.apache.jackrabbit.oak.plugins.document.persistentCache.PersistentCache)
	at org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.write(NodeCache.java:85)
	at org.apache.jackrabbit.oak.plugins.document.persistentCache.NodeCache.put(NodeCache.java:130)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.applyChanges(DocumentNodeStore.java:1060)
	at org.apache.jackrabbit.oak.plugins.document.Commit.applyToCache(Commit.java:599)
	at org.apache.jackrabbit.oak.plugins.document.CommitQueue.afterTrunkCommit(CommitQueue.java:127)
	- locked <0x0000000781890788> (a org.apache.jackrabbit.oak.plugins.document.CommitQueue)
	at org.apache.jackrabbit.oak.plugins.document.CommitQueue.done(CommitQueue.java:83)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.done(DocumentNodeStore.java:637)

To avoid blocking the foreground thread, one solution is to store all data in a separate thread. If there is too much data added, then some of the data is not stored. If possible, the data that was not referenced a lot, and / or old revisions of documents (if new revisions are available).

Issue Links

• blocks: OAK-3286 Persistent Cache improvements
• is duplicated by: OAK-2686 Persistent cache: log activity and timing data, and possible optimizations
• is duplicated by: OAK-2745 PersistentCache should rely on eviction callback to add entry to the persistent cache
• is related to: OAK-4882 Bottleneck in the asynchronous persistent cache
• is related to: OAK-7559 CacheActionDispatcher not memory bound
• relates to: OAK-2815 Persistent cache may block commit
• relates to: OAK-4123 Persistent cache: allow to configure the add data concurrency
• requires: OAK-3997 Include eviction cause to the LIRS removal callback