[OAK-3007] SegmentStore cache does not take "string" map into account

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.16, 1.3.3, 1.4
• Component/s: segmentmk
• Labels: doc-impacting resilience scalability
• Assignee: Michael Dürig
• Reporter: Thomas Mueller
• Created: 18/Jun/15 13:40
• Updated: 16/Feb/21 09:52
• Resolved: 15/Jul/15 14:08

Description

The SegmentStore cache size calculation ignores the size of the field Segment.string (a concurrent hash map). It looks like a regular segment in a memory mapped file has the size 1024, no matter how many strings are loaded in memory. This can lead to out of memory. There seems to be no way to limit (configure) the amount of memory used by strings. In one example, 100'000 segments are loaded in memory, and 5 GB are used for Strings in that map.

We need a way to configure the amount of memory used for that. This seems to be basically a cache. OAK-2688 does this, but it would be better to have one cache with a configurable size limit.

Issue Links

• is duplicated by: OAK-2688 Segment.readString optimization
• is related to: OAK-3075 Compaction Estimation should type check binary properties
• is related to: OAK-3889 SegmentMk StringCache memory leak
• is related to: OAK-3089 LIRS cache: zero size cache causes IllegalArgumentException
• is related to: OAK-3055 Improve segment cache in SegmentTracker
• relates to: OAK-3109 OOME in tarkmk standby tests