[LUCENE-328] Some utilities for a compact sparse filter

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: core/search

Environment

Operating System: other
Platform: Other

• Bugzilla Id: 32921
• Assignee: Unassigned
• Reporter: Paul Elschot
• Created: 04/Jan/05 00:38
• Updated: 28/Aug/22 11:20
• Resolved: 28/Jun/06 01:38

Description

Two files are attached that might form the basis for an alternative
filter implementation that is more memory efficient than one bit
per doc when less than about 1/8 of the docs pass through the filter.

The document numbers are stored in RAM as VInt's from the Lucene index
format. These VInt's encode the difference between two successive
document numbers, much like a PositionDelta in the Positions:
http://jakarta.apache.org/lucene/docs/fileformats.html

The getByteSize() method can be used to verify the compression
once a SortedVIntList is constructed.
The precise conditions under which this is more memory efficient than
one bit per document are not easy to specify in advance.

Issue Links

• is related to: LUCENE-584 Decouple Filter from BitSet