[LUCENE-6392] Add offset limit to Highlighter's TokenStreamFromTermVector

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.2
• Component/s: modules/highlighter
• Lucene Fields: New, Patch Available
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 03/Apr/15 17:30
• Updated: 28/Aug/22 14:30
• Resolved: 22/Apr/15 21:48

Description

The Highlighter's TokenStreamFromTermVector utility, typically accessed via TokenSources, should have the ability to filter out tokens beyond a configured offset. There is a TODO there already, and this issue addresses it. New methods in TokenSources now propagate a limit.

This patch also includes some memory saving optimizations, to be described shortly.

Issue Links

• depends upon: LUCENE-6423 New LimitTokenOffsetFilter
• is related to: LUCENE-6445 Highlighter TokenSources simplification; just one getAnyTokenStream()