[LUCENE-4479] TokenSources.getTokenStream() doesn't return correctly for termvectors with positions but no offsets

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0
• Fix Version/s: 4.1, 6.0
• Component/s: modules/highlighter
• Lucene Fields: New
• Assignee: Alan Woodward
• Reporter: Alan Woodward
• Created: 12/Oct/12 14:31
• Updated: 28/Aug/22 13:29
• Resolved: 20/Oct/12 19:39

Description

The javadocs for TokenSources.getTokenStream(Terms, boolean) state:

"Low level api. Returns a token stream or null if no offset info available
in index. This can be used to feed the highlighter with a pre-parsed token
stream"

However, if the Terms instance passed in has positions but no offsets stored, a TokenStream is incorrectly returned, rather than null.

This has the effect of incorrectly highlighting fields with term vectors and positions, but no offsets. All highlighting markup is prepended to the beginning of the field.