[LUCENE-5221] SimilarityBase.computeNorm is inconsistent with TFIDFSimilarity

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.4
• Fix Version/s: 4.5, 6.0
• Component/s: core/search
• Labels: normalize search similarity
• Lucene Fields: New
• Assignee: Unassigned
• Reporter: Yubin Kim
• Created: 17/Sep/13 16:05
• Updated: 23/Nov/24 04:44
• Resolved: 18/Sep/13 16:03

Description

SimilarityBase.computeNorm Javadoc indicates that the doc length should be encoded in the same way as TFIDFSimilarity. However, when discountOverlaps is false, what gets encoded is SmallFloat.floatToByte315((boost / (float) Math.sqrt(docLen / boost))); rather than SmallFloat.floatToByte315((boost / (float) Math.sqrt(length))); due to the extra / state.getBoost() term in SimilarityBase.computeNorm:

final float numTerms;
if (discountOverlaps)
numTerms = state.getLength() - state.getNumOverlap();
else
numTerms = state.getLength() / state.getBoost();
return encodeNormValue(state.getBoost(), numTerms);