Index: src/java/org/apache/lucene/search/Similarity.java
===================================================================
--- src/java/org/apache/lucene/search/Similarity.java (revision 813992)
+++ src/java/org/apache/lucene/search/Similarity.java (working copy)
@@ -29,17 +29,201 @@
import java.util.IdentityHashMap;
import java.util.Iterator;
-/** Expert: Scoring API.
- * Subclasses implement search scoring.
+/**
+ * Expert: Scoring API.
*
- *
The score of query q for document d correlates to the
- * cosine-distance or dot-product between document and query vectors in a
+ *
Similarity defines the components of Lucene scoring.
+ * Overriding computation of these components is a convenient
+ * way to alter Lucene scoring.
+ *
+ *
Suggested reading:
+ *
+ * Introduction To Information Retrieval, Chapter 6.
+ *
+ *
The following describes how Lucene scoring evolves from
+ * underlying information retrieval models to (efficient) implementation.
+ *
+ *
Lucene combines
+ *
+ * Boolean model (BM) of Information Retrieval
+ * with
*
- * Vector Space Model (VSM) of Information Retrieval.
- * A document whose vector is closer to the query vector in that model is scored higher.
+ * Vector Space Model (VSM) of Information Retrieval -
+ * documents "approved" by BM are scored by VSM.
*
- * The score is computed as follows:
+ *
In VSM, documents and queries are represented as
+ * weighted vectors in a multi-dimensional space,
+ * where each distinct index term is a dimension,
+ * and weights are
+ * Tf-idf values.
*
+ *
It should be noted that VSM does not require weights to be Tf-idf values,
+ * but Tf-idf values are believed to produce search results of high quality,
+ * and so Lucene is using Tf-idf.
+ * The implementation of Tf and Idf in Lucene is discussed in more detail below,
+ * but for now, for completion, let's just say that
+ * Tf(t,x) is proportional to the number of occurrences of term t in x
+ * (x is either a document or a query) and
+ * idf(t) is proportional to the inverse of the
+ * number of index documents containing term t.
+ *
+ *
VSM score of document d for query q is the
+ *
+ * Cosine Similarity
+ * of the weighted query vectors V(q) and V(d):
+ *
+ *
+ *
+ *
+ * |
+ * cosine-similarity(q,d) =
+ * |
+ *
+ *
+ * | V(q) · V(d) |
+ * | ––––––––– |
+ * | |V(q)| |V(d)| |
+ *
+ * |
+ *
+ *
+ *
+ *
+ * Where V(q) · V(d) is the
+ * dot product
+ * of the weighted vectors,
+ * and |V(q)| and |V(d)| are their
+ * Euclidean norms.
+ *
+ * Note: the above equation can be viewed as the dot product of
+ * the normalized weighted vectors, in the sense that dividing
+ * V(q) by its euclidean norm is normalizing it to a unit vector.
+ *
+ *
This is, in principle, the scoring function of Lucene.
+ * However there are a few refinements, for both search quality and usability:
+ *
+ * - Normalizing V(d) to the unit vector is known to be problematic in that
+ * it removes all document length information.
+ * For some documents removing this info is probably ok,
+ * e.g. a document made by duplicating a certain paragraph 10 times,
+ * especially if that paragraph is made of distinct terms.
+ * But for a document which contains no duplicated paragraphs,
+ * this might be wrong.
+ * To avoid this problem, a different document length normalization
+ * factor is used, which normalizes to a vector equal to or larger
+ * than the unit vector: doc-len-norm(d).
+ *
+ *
+ * - At indexing, users can specify that certain documents are more
+ * important than others, by assigning a document boost.
+ * For this, the score of each document is also multiplied by its boost value
+ * doc-boost(d).
+ *
+ *
+ * - Lucene is field based, hence each query term applies to a single
+ * field, document length normalization is by the length of the certain field,
+ * and in addition to document boost there are also document fields boosts.
+ *
+ *
+ * - The same field can be added to a document during indexing several times,
+ * and so the boost of that field is the multiplication of the boosts of
+ * the separate additions (or parts) of that field within the document.
+ *
+ *
+ * - At search time users can specify boosts to each query, sub-query, and
+ * each query term, hence the contribution of a query term to the score of
+ * a document is multiplied by the boost of that query term query-boost(q).
+ *
+ *
+ * - A document may match a multi term query without containing all
+ * the terms of that query (this is correct for some of the queries),
+ * and users can further reward documents matching more query terms
+ * through a coordination factor, which is usually larger when
+ * more terms are matched: coord-factor(q,d).
+ *
+ *
+ *
+ * Under the simplifying assumption of a single field in the index,
+ * here is the resulted scoring formula:
+ *
+ *
+ *
+ *
+ * |
+ * score(q,d) =
+ * coord-factor(q,d) ·
+ * query-boost(q) ·
+ * |
+ *
+ *
+ * | V(q) · V(d) |
+ * | ––––––––– |
+ * | |V(q)| |
+ *
+ * |
+ *
+ * · doc-len-norm(d)
+ * · doc-boost(d)
+ * |
+ *
+ *
+ *
+ *
+ * The above formula is a simplification in the sense that (1) terms and documents
+ * are fielded and (2) boosts are usually per query term rather than per query.
+ *
+ *
For efficient score computation some scoring components
+ * are computed and aggregated in advance:
+ *
+ *
+ * - Query-boost for the query (actually for each query term)
+ * is known when search starts.
+ *
+ *
+ * - Query Euclidean norm |V(q)| can be computed when search starts,
+ * as it is independent of the document being scored.
+ * From search optimization perspective, it is a valid question
+ * why bother to normalize the query at all, because all
+ * scored documents will be multiplied by the same |V(q)|,
+ * and hence documents ranks (their order by score) will not
+ * be affected by this normalization.
+ * There are two good reasons to keep this normalization:
+ *
+ * - Recall that
+ *
+ * Cosine Similarity can be used find how similar
+ * two documents are. One can use Lucene for e.g.
+ * clustering, and use a document as a query to compute
+ * its similarity to other documents.
+ * In this use case it is important that the score of document d3
+ * for query d1 is comparable to the score of document d3
+ * for query d2. In other words, scores of a document for two
+ * distinct queries should be comparable.
+ * There are other applications that may require this.
+ * And this is exactly what normalizing the query vector V(q)
+ * provides: comparability (to a certain extent) of two or more queries.
+ *
+ *
+ * - Applying query normalization on the scores helps to keep the
+ * scores around the unit vector, hence preventing loss of score data
+ * because of floating point precision limitations.
+ *
+ *
+ *
+ *
+ * - Document length norm doc-len-norm(d) and document
+ * boost doc-boost(d) are known at indexing time.
+ * They are computed in advance and their multiplication
+ * is saved as a single value in the index: norm(d).
+ * (In the equations below, norm(t in d) means norm(field(t) in doc d)
+ * where field(t) is the field associated with term t.)
+ *
+ *
+ *
+ * Lucene's actual scoring function is derived from the above.
+ * The color codes demonstrate how actual scoring function components
+ * relate to those of the theoretical scoring function:
+ *
*
*
* |
@@ -47,18 +231,18 @@
* |
* |
* score(q,d) =
- * coord(q,d) ·
- * queryNorm(q) ·
+ * coord(q,d) ·
+ * queryNorm(q) ·
* |
*
* ∑
* |
*
* (
- * tf(t in d) ·
- * idf(t)2 ·
- * t.getBoost() ·
- * norm(t,d)
+ * tf(t in d) ·
+ * idf(t)2 ·
+ * t.getBoost() ·
+ * norm(t,d)
* )
* |
*
@@ -75,10 +259,14 @@
*
* -
*
- * tf(t in d)
+ * tf(t in d)
* correlates to the term's frequency,
* defined as the number of times term t appears in the currently scored document d.
* Documents that have more occurrences of a given term receive a higher score.
+ * Note that tf(t in q) is assumed to be 1 and therefore it does not appear in this equation,
+ * However if a query contains twice the same term, there will be
+ * two term-queries with that same term and hence the computation would still be correct (although
+ * not very efficient).
* The default computation for tf(t in d) in
* {@link org.apache.lucene.search.DefaultSimilarity#tf(float) DefaultSimilarity} is:
*
@@ -98,10 +286,12 @@
*
*
-
*
- * idf(t) stands for Inverse Document Frequency. This value
+ * idf(t) stands for Inverse Document Frequency. This value
* correlates to the inverse of docFreq
* (the number of documents in which the term t appears).
* This means rarer terms give higher contribution to the total score.
+ * idf(t) appears for t in both the query and the document,
+ * hence it is squared in the equation.
* The default computation for idf(t) in
* {@link org.apache.lucene.search.DefaultSimilarity#idf(int, int) DefaultSimilarity} is:
*
@@ -131,7 +321,7 @@
*
*
-
*
- * coord(q,d)
+ * coord(q,d)
* is a score factor based on how many of the query terms are found in the specified document.
* Typically, a document that contains more of the query's terms will receive a higher score
* than another document with fewer query terms.
@@ -143,7 +333,7 @@
*
*
-
*
- * queryNorm(q)
+ * queryNorm(q)
*
* is a normalizing factor used to make scores between queries comparable.
* This factor does not affect document ranking (since all ranked documents are multiplied by the same factor),
@@ -152,7 +342,7 @@
*
* The default computation in
* {@link org.apache.lucene.search.DefaultSimilarity#queryNorm(float) DefaultSimilarity}
- * is:
+ * produces a Euclidean norm:
*
*
*
@@ -209,7 +399,7 @@
*
* -
*
- * t.getBoost()
+ * t.getBoost()
* is a search time boost of term t in the query q as
* specified in the query text
* (see query syntax),
@@ -225,7 +415,7 @@
*
*
-
*
- * norm(t,d) encapsulates a few (indexing time) boost and length factors:
+ * norm(t,d) encapsulates a few (indexing time) boost and length factors:
*
*
* - Document boost - set by calling
@@ -277,8 +467,8 @@
* {@link org.apache.lucene.store.Directory directory} and
* {@link #decodeNorm(byte) decoded} back to a float norm value.
* This encoding/decoding, while reducing index size, comes with the price of
- * precision loss - it is not guaranteed that decode(encode(x)) = x.
- * For instance, decode(encode(0.89)) = 0.75.
+ * precision loss - it is not guaranteed that decode(encode(x)) = x.
+ * For instance, decode(encode(0.89)) = 0.75.
* Also notice that search time is too late to modify this norm part of scoring, e.g. by
* using a different {@link Similarity} for search.
*