[LUCENE-6423] New LimitTokenOffsetFilter

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.2
• Component/s: modules/analysis
• Lucene Fields: New
• Assignee: David Smiley
• Reporter: David Smiley
• Created: 14/Apr/15 19:41
• Updated: 28/Aug/22 14:31
• Resolved: 22/Apr/15 19:51

Description

It would be nice to have a token filter that limited based on the offset. I suggest the start offset. It should be named LimitTokenOffsetFilter to have a name similar to the other LimitToken*Filter choices, including a "consumeAllTokens" option. I plan to use this filter in LUCENE-6392 (to limit tokens from analyzed text for applicable methods in TokenSources) and in LUCENE-6375.

Issue Links

• is depended upon by: LUCENE-6375 Inconsistent interpretation of maxDocCharsToAnalyze in Highlighter & WeightedSpanTermExtractor
• is depended upon by: LUCENE-6392 Add offset limit to Highlighter's TokenStreamFromTermVector
• relates to: LUCENE-6445 Highlighter TokenSources simplification; just one getAnyTokenStream()