[LUCENE-1737] Always use bulk-copy when merging stored fields and term vectors

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.1, 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 09/Jul/09 18:47
• Updated: 28/Aug/22 12:03
• Resolved: 19/Dec/10 10:18

Description

Lucene has nice optimizations in place during merging of stored fields
(LUCENE-1043) and term vectors (LUCENE-1120) whereby the bytes are
bulk copied to the new segmetn. This is much faster than decoding &
rewriting one document at a time.

However the optimization is rather brittle: it relies on the mapping
of field name to number to be the same ("congruent") for the segment
being merged.

Unfortunately, the field mapping will be congruent only if the app
adds the same fields in precisely the same order to each document.

I think we should fix IndexWriter to assign the same field number for
a given field that has been assigned in the past. Ie, when writing a
new segment, we pre-seed the field numbers based on past segments.
All other aspects of FieldInfo would remain fully dynamic.

Issue Links

• is blocked by: LUCENE-2814 stop writing shared doc stores across segments