[SOLR-342] Add support for Lucene's new Indexing and merge features (excluding Document/Field/Token reuse)

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Component/s: update
• Assignee: Grant Ingersoll
• Reporter: Grant Ingersoll
• Created: 20/Aug/07 20:52
• Updated: 05/Mar/08 20:23
• Resolved: 05/Mar/08 20:23

Description

LUCENE-843 adds support for new indexing capabilities using the setRAMBufferSizeMB() method that should significantly speed up indexing for many applications. To fix this, we will need trunk version of Lucene (or wait for the next official release of Lucene)

Side effect of this is that Lucene's new, faster StandardTokenizer will also be incorporated.

Also need to think about how we want to incorporate the new merge scheduling functionality (new default in Lucene is to do merges in a background thread)

Issue Links

• is blocked by: LUCENE-1173 index corruption autoCommit=false
• is blocked by: LUCENE-1168 TermVectors index files can become corrupt when autoCommit=false
• is blocked by: LUCENE-1175 occasional MergeException while indexing
• is blocked by: LUCENE-1176 TermVectors corruption case when autoCommit=false
• is blocked by: LUCENE-1177 IW.optimize() can do too many merges at the very end
• relates to: SOLR-403 Utilize Lucene 2.3's new Document/Field/Token reuse capabilities
• relates to: LUCENE-843 improve how IndexWriter uses RAM to buffer added documents