[LUCENE-3091] Highlighter's TokenSources should use TermVector positions if indexed

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: modules/highlighter
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Michael McCandless
• Created: 12/May/11 17:07
• Updated: 28/Aug/22 12:46

Description

Spinoff from LUCENE-3087

TokenSources is able to use positions indexed into TermVectors, if you pass "false" for the tokenPositionsGuaranteedContiguous argument to getTokenStream.

But if you pass true, it doesn't use the positions. Maybe, we are not allowed (supposed) to pass true if the posIncr could be 0? But, if that is OK/allowed, then we should fix the code to handle the posIncr=0 case correctly.

LUCENE-3087 made one simple fix, to guess at the posInc (1 or 0) based on the offsets of adjacent tokens.