[LUCENE-3218] Make CFS appendable

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4, 4.0-ALPHA
• Fix Version/s: 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New, Patch Available
• Assignee: Simon Willnauer
• Reporter: Simon Willnauer
• Created: 20/Jun/11 07:52
• Updated: 10/May/13 10:42
• Resolved: 04/Nov/11 22:07

Description

Currently CFS is created once all files are written during a flush / merge. Once on disk the files are copied into the CFS format which is basically a unnecessary for some of the files. We can at any time write at least one file directly into the CFS which can save a reasonable amount of IO. For instance stored fields could be written directly during indexing and during a Codec Flush one of the written files can be appended directly. This optimization is a nice sideeffect for lucene indexing itself but more important for DocValues and LUCENE-3216 we could transparently pack per field files into a single file only for docvalues without changing any code once LUCENE-3216 is resolved.

Issue Links

• incorporates: LUCENE-3201 improved compound file handling
• is depended upon by: LUCENE-3216 Store DocValues per segment instead of per field