[LUCENE-7457] Default doc values format should optimize for iterator access

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 7.0
• Lucene Fields: New
• Assignee: Adrien Grand
• Reporter: Michael McCandless
• Created: 21/Sep/16 17:52
• Updated: 28/Aug/22 15:03
• Resolved: 03/Oct/16 07:38

Description

In LUCENE-7407 we switched doc values consumption from random access API to an iterator API, but nothing was done there to improve the codec. We should do that here.

At a bare minimum we should fix the existing very-sparse case to be a true iterator, and not wrapped with the silly legacy wrappers.

I think we should also increase the threshold (currently 1%?) when we switch from dense to sparse encoding. This should fix LUCENE-7253, making merging of sparse doc values efficient ("pay for what you use").

I'm sure there are many other things to explore to let codecs "take advantage" of the fact that they no longer need to offer random access to doc values.

Issue Links

• is related to: LUCENE-7407 Explore switching doc values to an iterator API
• supercedes: LUCENE-7253 Make sparse doc values and segments merging more efficient