[OAK-2247] CopyOnWriteDirectory implementation for Lucene for use in indexing

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.15, 1.2.3, 1.3.0, 1.4
• Component/s: lucene
• Labels: docs-impacting performance
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 30/Oct/14 05:05
• Updated: 08/Dec/16 07:00
• Resolved: 22/May/15 07:13

Description

Currently a Lucene index when is written directly to OakDirectory. For reindex case it might happen that Lucene merge policy read the written index files again and then perform a sgement merge. This might have lower performance when OakDirectroy is writing to remote storage.

Instead of that we can implement a CopyOnWriteDirectory on similar lines to OAK-1724 where CopyOnReadDirectory support copies the index locally for faster access.

At high level flow would be

1. While writing index the index file is first written to local directory
2. Any write is done locally and once a file is written its written asynchronously to OakDirectory
3. When IndexWriter is closed it would wait untill all the write is completed

This needs to be benchmarked with existing reindex timings to see it its actually beneficial

Issue Links

• is related to: OAK-4024 CoW uses incorrect directory on re-indexing when indexPath property is used
• is related to: OAK-5238 IndexCopier causes concurrent update on NodeBuilder
• is related to: OAK-3609 Enable CopyOnWrite by default