[LUCENE-2632] FilteringCodec, TeeCodec, TeeDirectory

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Andrzej Bialecki
• Created: 02/Sep/10 15:32
• Updated: 28/Aug/22 12:32

Description

This issue adds two new Codec implementations:

• TeeCodec: there have been attempts in the past to implement parallel writing to multiple indexes so that they are all synchronized. This was however complicated due to the complexity of IndexWriter/SegmentMerger logic. The solution presented here offers a similar functionality but working on a different level - as the name suggests, the TeeCodec duplicates index data into multiple output Directories.

• TeeDirectory (used also in TeeCodec) is a simple abstraction to perform Directory operations on several directories in parallel (effectively mirroring their data). Optionally it's possible to specify a set of suffixes of files that should be mirrored so that non-matching files are skipped.

• FilteringCodec is related in a remote way to the ideas of index pruning presented in LUCENE-1812 and the concept of tiered search. Since we can use TeeCodec to write to multiple output Directories in a synchronized way, we could also filter out or modify some of the data that is being written. The FilteringCodec provides this functionality, so that you can use like this:

  IndexWriter --> TeeCodec
                   |  |
                   |  +--> StandardCodec --> Directory1
                   +--> FilteringCodec --> StandardCodec --> Directory2
  

The end result of this chain is two indexes that are kept in sync - one is the full regular index, and the other one is a filtered index.