[LUCENE-4283] Support more frequent skip with Block Postings Format

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Later
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Han Jiang
• Created: 02/Aug/12 10:32
• Updated: 28/Aug/22 13:23
• Resolved: 21/Aug/12 12:22

Description

This change works on the new bulk branch.

Currently, our BlockPostingsFormat only supports skipInterval==blockSize. Every time the skipper reaches the last level 0 skip point, we'll have to decode a whole block to read doc/freq data. Also, a higher level skip list will be created only for those df>blockSize^k, which means for most terms, skipping will just be a linear scan. If we increase current blockSize for better bulk i/o performance, current skip setting will be a bottleneck.

For ForPF, the encoded block can be easily splitted if we set skipInterval=32*k.

Issue Links

• is related to: LUCENE-3892 Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
• relates to: LUCENE-4225 New FixedPostingsFormat for less overhead than SepPostingsFormat