[LUCENE-4069] Segment-level Bloom filters

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.6, 4.0-ALPHA
• Fix Version/s: 4.0-BETA, 6.0
• Component/s: core/index
• Lucene Fields: New, Patch Available
• Assignee: Mark Harwood
• Reporter: Mark Harwood
• Created: 18/May/12 15:44
• Updated: 28/Aug/22 13:17
• Resolved: 02/Aug/12 13:25

Description

An addition to each segment which stores a Bloom filter for selected fields in order to give fast-fail to term searches, helping avoid wasted disk access.

Best suited for low-frequency fields e.g. primary keys on big indexes with many segments but also speeds up general searching in my tests.

Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments

Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU

Patch based on 3.6 codebase attached.
There are no 3.6 API changes currently - to play just add a field with "_blm" on the end of the name to invoke special indexing/querying capability. Clearly a new Field or schema declaration would need adding to APIs to configure the service properly.

Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

Issue Links

• relates to: SOLR-1375 BloomFilter on a field