[LUCENE-843] improve how IndexWriter uses RAM to buffer added documents

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2
• Fix Version/s: 2.3
• Component/s: core/index
• Lucene Fields: New, Patch Available
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 22/Mar/07 17:05
• Updated: 28/Aug/22 11:36
• Resolved: 12/Aug/07 15:48

Description

I'm working on a new class (MultiDocumentWriter) that writes more than
one document directly into a single Lucene segment, more efficiently
than the current approach.

This only affects the creation of an initial segment from added
documents. I haven't changed anything after that, eg how segments are
merged.

The basic ideas are:

• Write stored fields and term vectors directly to disk (don't
  use up RAM for these).

• Gather posting lists & term infos in RAM, but periodically do
  in-RAM merges. Once RAM is full, flush buffers to disk (and
  merge them later when it's time to make a real segment).

• Recycle objects/buffers to reduce time/stress in GC.

• Other various optimizations.

Some of these changes are similar to how KinoSearch builds a segment.
But, I haven't made any changes to Lucene's file format nor added
requirements for a global fields schema.

So far the only externally visible change is a new method
"setRAMBufferSize" in IndexWriter (and setMaxBufferedDocs is
deprecated) so that it flushes according to RAM usage and not a fixed
number documents added.

Issue Links

• is blocked by: LUCENE-845 If you "flush by RAM usage" then IndexWriter may over-merge
• is related to: SOLR-342 Add support for Lucene's new Indexing and merge features (excluding Document/Field/Token reuse)