[LUCENE-9378] Configurable compression for BinaryDocValues

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.8
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Viral Gandhi
• Created: 21/May/20 16:57
• Updated: 16/Jun/21 08:28
• Resolved: 12/Nov/20 21:42

Description

Lucene 8.5.1 includes a change to always compress BinaryDocValues. This caused (~30%) reduction in our red-line QPS (throughput). 

We think users should be given some way to opt-in for this compression feature instead of always being enabled which can have a substantial query time cost as we saw during our upgrade. Michael McCandless suggested one possible approach by introducing a mode in Lucene80DocValuesFormat (COMPRESSED and UNCOMPRESSED) and allowing users to create a custom Codec subclassing the default Codec and pick the format they want.

Idea is similar to Lucene50StoredFieldsFormat which has two modes, Mode.BEST_SPEED and Mode.BEST_COMPRESSION.

Here's related issues for adding benchmark covering BINARY doc values query-time performance - https://github.com/mikemccand/luceneutil/issues/61

Issue Links

• is caused by: LUCENE-9211 Adding compression to BinaryDocValues storage
• links to: GitHub Pull Request #1543
• links to: GitHub Pull Request #2069