[LUCENE-9211] Adding compression to BinaryDocValues storage

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 8.5
• Component/s: core/codecs
• Labels: pull-request-available
• Lucene Fields: New
• Assignee: Mark Harwood
• Reporter: Mark Harwood
• Created: 07/Feb/20 09:44
• Updated: 22/Nov/24 14:43
• Resolved: 18/Feb/20 14:44

Description

While SortedSetDocValues can be used today to store identical values in a compact form this is not effective for data with many unique values.

The proposal is that BinaryDocValues should be stored in LZ4 compressed blocks which can dramatically reduce disk storage costs in many cases. The proposal is blocks of a number of documents are stored as a single compressed blob along with metadata that records offsets where the original document values can be found in the uncompressed content.

There's a trade-off here between efficient compression (more docs-per-block = better compression) and fast retrieval times (fewer docs-per-block = faster read access for single values). A fixed block size of 32 docs seems like it would be a reasonable compromise for most scenarios.

A PR is up for review here https://github.com/apache/lucene-solr/pull/1234

Issue Links

• causes: LUCENE-9378 Configurable compression for BinaryDocValues
• links to: GitHub Pull Request #1234
• links to: GitHub Pull Request #1264