[LUCENE-2312] Search on IndexWriter's RAM Buffer

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0-ALPHA
• Component/s: core/search
• Lucene Fields: New
• Assignee: Michael Busch
• Reporter: Jason Rutherglen
• Created: 12/Mar/10 23:09
• Updated: 28/Aug/22 12:21

Description

In order to offer user's near realtime search, without incurring
an indexing performance penalty, we can implement search on
IndexWriter's RAM buffer. This is the buffer that is filled in
RAM as documents are indexed. Currently the RAM buffer is
flushed to the underlying directory (usually disk) before being
made searchable.

Todays Lucene based NRT systems must incur the cost of merging
segments, which can slow indexing.

Michael Busch has good suggestions regarding how to handle deletes using max doc ids.
https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12841923&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12841923

The area that isn't fully fleshed out is the terms dictionary,
which needs to be sorted prior to queries executing. Currently
IW implements a specialized hash table. Michael B has a
suggestion here:
https://issues.apache.org/jira/browse/LUCENE-2293?focusedCommentId=12841915&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12841915

Issue Links

• incorporates: LUCENE-2575 Concurrent byte and int block implementations
• incorporates: LUCENE-3199 Add non-desctructive sort to BytesRefHash
• incorporates: LUCENE-3399 Enable replace-able field caches
• is blocked by: LUCENE-2324 Per thread DocumentsWriters that write their own private segments
• is related to: CASSANDRA-2915 Lucene based Secondary Indexes
• relates to: LUCENE-2346 Explore other in-memory postinglist formats for realtime search