[LUCENE-5729] explore random-access methods to IndexInput

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Robert Muir
• Created: 03/Jun/14 00:25
• Updated: 28/Aug/22 14:09
• Resolved: 04/Jun/14 18:51

Description

Traditionally lucene access is mostly reading lists of postings and geared at that, but for random-access stuff like docvalues, it just creates overhead.

So today we are hacking around it, by doing this random access with seek+readXXX, but this is inefficient (additional checks by the jdk that we dont need).

As a hack, I added the following to IndexInput, changed direct packed ints decode to use them, and implemented in MMapDir:

byte readByte(long pos) --> ByteBuffer.get(pos)
short readShort(long pos) --> ByteBuffer.getShort(pos)
int readInt(long pos) --> ByteBuffer.getInt(pos)
long readLong(long pos) --> ByteBuffer.getLong(pos)

This gives ~30% performance improvement for docvalues (numerics, sorting strings, etc)

We should do a few things first before working this (LUCENE-5728: use slice api in decode, pad packed ints so we only have one i/o call ever, etc etc) but I think we need to figure out such an API.

It could either be on indexinput like my hack (this is similar to ByteBuffer API with both relative and absolute methods), or we could have a separate API. But i guess arguably IOContext exists to supply hints too, so I dont know which is the way to go.