[LUCENE-2723] Speed up Lucene's low level bulk postings read API

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 4.1
• Component/s: core/index
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 25/Oct/10 19:13
• Updated: 28/Aug/22 12:34
• Resolved: 11/Sep/12 16:23

Description

Spinoff from LUCENE-1410.

The flex DocsEnum has a simple bulk-read API that reads the next chunk
of docs/freqs. But it's a poor fit for intblock codecs like FOR/PFOR
(from LUCENE-1410). This is not unlike sucking coffee through those
tiny plastic coffee stirrers they hand out airplanes that,
surprisingly, also happen to function as a straw.

As a result we see no perf gain from using FOR/PFOR.

I had hacked up a fix for this, described at in my blog post at
http://chbits.blogspot.com/2010/08/lucene-performance-with-pfordelta-codec.html

I'm opening this issue to get that work to a committable point.

So... I've worked out a new bulk-read API to address performance
bottleneck. It has some big changes over the current bulk-read API:

• You can now also bulk-read positions (but not payloads), but, I
  have yet to cutover positional queries.

• The buffer contains doc deltas, not absolute values, for docIDs
  and positions (freqs are absolute).

• Deleted docs are not filtered out.

• The doc & freq buffers need not be "aligned". For fixed intblock
  codecs (FOR/PFOR) they will be, but for varint codecs (Simple9/16,
  Group varint, etc.) they won't be.

It's still a work in progress...

Issue Links

• is blocked by: LUCENE-2694 MTQ rewrite + weight/scorer init should be single pass
• relates to: LUCENE-2890 Track down performance regression on BulkPostings