[OAK-3132] Reduce memory usage of OakIndexFile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Component/s: lucene
• Assignee: Unassigned
• Reporter: Marcel Reutegger
• Created: 22/Jul/15 07:26
• Updated: 04/Oct/17 17:46
• Resolved: 04/Oct/17 17:46

Description

OakIndexFile splits an index file into Blob chunks and keeps one of the chunks in memory at a time. The chunk size is 32k by default but can be controlled with a property on the index definition.

In a test repository I observed OakIndexFile instances consuming more than 600MB of heap, with a chunk size of 1MB. This seems quite high for a system where no query was active at that time.

I think we should look into alternative ways to read from the Blob than just plain old InputStream. E.g. with a ByteChannel we could push the caching aspect to the Blob implementation, which could leverage an OS buffer cache.

Issue Links

• relates to: OAK-6269 Support non chunk storage in OakDirectory