[OAK-9052] Reindexing using --doc-traversal-mode may need a lot of memory

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.30.0
• Component/s: indexing, mongomk, run
• Assignee: Thomas Mueller
• Reporter: Thomas Mueller
• Created: 05/May/20 13:09
• Updated: 27/May/20 06:19
• Resolved: 12/May/20 06:57

Description

Indexing using oak-run and --doc-traversal-mode uses the FlatFileStore. For aggregation, there is a limit on memory usage, by default around 100 MB. Depending on the content structure, this limit can be exceeded.

It would be good to find a way to avoid a memory limit, for example using a temporary storage (a file, or a persistent key/value store).

Issue Links

• is cloned by: OAK-9093 Reindexing using --doc-traversal-mode may log too much
• is related to: OAK-7284 Reindexing using --doc-traversal-mode can hit ConcurrentModificationException during aggregation
• relates to: OAK-9088 FlatFileStoreIteratorTest fails on Windows