-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 8.8
-
Component/s: None
-
Labels:None
-
Lucene Fields:New
Lucene 8.5.1 includes a change to always compress BinaryDocValues. This caused (~30%) reduction in our red-line QPS (throughput).
We think users should be given some way to opt-in for this compression feature instead of always being enabled which can have a substantial query time cost as we saw during our upgrade. Michael McCandless suggested one possible approach by introducing a mode in Lucene80DocValuesFormat (COMPRESSED and UNCOMPRESSED) and allowing users to create a custom Codec subclassing the default Codec and pick the format they want.
Idea is similar to Lucene50StoredFieldsFormat which has two modes, Mode.BEST_SPEED and Mode.BEST_COMPRESSION.
Here's related issues for adding benchmark covering BINARY doc values query-time performance - https://github.com/mikemccand/luceneutil/issues/61
- is caused by
-
LUCENE-9211 Adding compression to BinaryDocValues storage
-
- Closed
-
- links to