[LUCENE-3069] Lucene should have an entirely memory resident term dictionary

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0-ALPHA
• Fix Version/s: 4.7
• Component/s: core/index, core/search
• Labels: gsoc2013
• Lucene Fields: New
• Assignee: Han Jiang
• Reporter: Simon Willnauer
• Created: 04/May/11 14:09
• Updated: 10/Jan/20 15:04
• Resolved: 16/Mar/14 07:06

Description

FST based TermDictionary has been a great improvement yet it still uses a delta codec file for scanning to terms. Some environments have enough memory available to keep the entire FST based term dict in memory. We should add a TermDictionary implementation that encodes all needed information for each term into the FST (custom fst.Output) and builds a FST from the entire term not just the delta.

Issue Links

• is related to: LUCENE-8285 FSTOrdPostingsFormat ironically does not support ord()
• is related to: LUCENE-3209 Memory codec

Sub-Tasks

• factor out a generic 'TermState' for better sharing in FST-based term dict: Closed
• Refactoring on PostingsWriterBase for delta-encoding: Closed