[LUCENE-1420] Similarity.lengthNorm and positionIncrement=0

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9
• Fix Version/s: 2.9
• Component/s: core/index
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Andrzej Bialecki
• Created: 13/Oct/08 10:08
• Updated: 28/Aug/22 11:54
• Resolved: 03/Nov/08 18:05

Description

Calculation of lengthNorm factor should in some cases take into account the number of tokens with positionIncrement=0. This should be made optional, to support two different scenarios:

• when analyzers insert artificially constructed tokens into TokenStream (e.g. ASCII-fied versions of accented terms, stemmed terms), and it's unlikely that users submit queries containing both versions of tokens: in this case lengthNorm calculation should ignore the tokens with positionIncrement=0.

• when analyzers insert synonyms, and it's likely that users may submit queries that contain multiple synonymous terms: in this case the lengthNorm should be calculated as it is now, i.e. it should take into account all terms no matter what is their positionIncrement.

The default should be backward-compatible, i.e. it should count all tokens.

(See also the discussion here: http://markmail.org/message/vfvmzrzhr6pya22h )