Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
Currently a Lucene index when is written directly to OakDirectory. For reindex case it might happen that Lucene merge policy read the written index files again and then perform a sgement merge. This might have lower performance when OakDirectroy is writing to remote storage.
Instead of that we can implement a CopyOnWriteDirectory on similar lines to OAK-1724 where CopyOnReadDirectory support copies the index locally for faster access.
At high level flow would be
- While writing index the index file is first written to local directory
- Any write is done locally and once a file is written its written asynchronously to OakDirectory
- When IndexWriter is closed it would wait untill all the write is completed
This needs to be benchmarked with existing reindex timings to see it its actually beneficial