[JENA-2354] OOM / increased memory consumption with TDB 4.9.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Jena 4.9.0
• Fix Version/s: Jena 4.10.0
• Component/s: TDB2
• Assignee: Andy Seaborne
• Reporter: Damien Obrist
• Created: 03/Aug/23 12:13
• Updated: 21/Jun/24 14:29
• Resolved: 28/Aug/23 20:18

Description

I have observed that after upgrading from TDB 4.8.0 to 4.9.0, memory consumption seems to have increased substantially.

Our application creates many TDB datasets, which are queried and updated concurrently. To profile the memory increase, I have implemented a small script, which creates up to 50 of our application-level objects, one after another. Each of these objects is backed by a TDB dataset under the hood, which gets created from scratch and filled with data (around 30k triples, 200MB folder size).

The following shows the memory profiles when running the script against our application for both TDB 4.8.0 / 4.9.0 and using 1G of heap space.

TDB 4.8.0

Memory increases only slightly and the script completes the creation of all 50 objects / TDB datasets:

TDB 4.9.0

Memory increases fast and the application crashes with an OOM after having created 8 objects / TDB datasets, during the creation of the 9th:

Stack trace:

java.lang.OutOfMemoryError: Java heap space
    at com.github.benmanes.caffeine.cache.FrequencySketch.ensureCapacity(FrequencySketch.java:92)
    at com.github.benmanes.caffeine.cache.SSMS.<init>(Unknown Source)
    at java.base/java.lang.invoke.LambdaForm$DMH/0x00000008028f7c40.newInvokeSpecial(LambdaForm$DMH)
    at java.base/java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(DelegatingMethodHandle$Holder)
    at java.base/java.lang.invoke.LambdaForm$MH/0x00000008028f7040.invoke_MT(LambdaForm$MH)
    at com.github.benmanes.caffeine.cache.LocalCacheFactory.loadFactory(LocalCacheFactory.java:86)
    at com.github.benmanes.caffeine.cache.LocalCacheFactory.newBoundedLocalCache(LocalCacheFactory.java:40)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3947)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache.<init>(BoundedLocalCache.java:3943)
    at com.github.benmanes.caffeine.cache.Caffeine.build(Caffeine.java:1051)
    at org.apache.jena.atlas.lib.cache.CacheCaffeine.<init>(CacheCaffeine.java:62)
    at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:45)
    at org.apache.jena.atlas.lib.CacheFactory.createCache(CacheFactory.java:33)
    at org.apache.jena.tdb2.store.nodetable.NodeTableCache.createCache(NodeTableCache.java:101)
    at org.apache.jena.tdb2.store.nodetable.NodeTableCache.<init>(NodeTableCache.java:95)
    at org.apache.jena.tdb2.store.nodetable.NodeTableCache.create(NodeTableCache.java:87)
    at org.apache.jena.tdb2.store.TDB2StorageBuilder.addNodeTableCache(TDB2StorageBuilder.java:302)
    at org.apache.jena.tdb2.store.TDB2StorageBuilder.buildNodeTable(TDB2StorageBuilder.java:287)
    at org.apache.jena.tdb2.store.TDB2StorageBuilder.buildStorage(TDB2StorageBuilder.java:189)
    at org.apache.jena.tdb2.store.TDB2StorageBuilder.build(TDB2StorageBuilder.java:97)
    at org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:95)
    at org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:63)
    at org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:139)
    at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:86)
    at org.apache.jena.tdb2.sys.DatabaseConnection.build(DatabaseConnection.java:112)
    at org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:74)
    at org.apache.jena.tdb2.sys.DatabaseConnection$$Lambda$8502/0x00000008028e3c40.apply(Unknown Source)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
    at org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:74)
    at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:63)
    at org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:54)
    at org.apache.jena.tdb2.DatabaseMgr.DB_ConnectCreate(DatabaseMgr.java:41) 

Issue Links

• links to: GitHub Pull Request #1996
• links to: GitHub Pull Request #2524