[LUCENE-7038] QueryScorer.init returns empty TokenStream if setMaxDocCharsToAnalyze is not previously called

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 5.4
• Component/s: modules/highlighter
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Jeff Stein
• Created: 19/Feb/16 19:43
• Updated: 28/Aug/22 14:50

Description

This is a regression since Lucene 4.10 regarding The QueryScorer class in the Highlighter module.

In 4.10, the QueryScorer.init method returns a working tokenStream even if the maxCharsToAnalyze variable is set to zero. In both versions, zero is the default value and in 4.10 it indicated that the entire stream should be returned, not an empty stream.

The problem is with the WeightedSpanTermExtractor always wrapping the tokenStream in a OffsetLimitTokenFilter filter, even when the passed down maxDocCharsToAnalyze variable is zero.

Issue Links

• is related to: LUCENE-6375 Inconsistent interpretation of maxDocCharsToAnalyze in Highlighter & WeightedSpanTermExtractor