Index: src/java/org/apache/lucene/search/Similarity.java
===================================================================
--- src/java/org/apache/lucene/search/Similarity.java (revision 813992)
+++ src/java/org/apache/lucene/search/Similarity.java (working copy)
@@ -29,17 +29,203 @@
import java.util.IdentityHashMap;
import java.util.Iterator;
-/** Expert: Scoring API.
- * Subclasses implement search scoring.
- *
- *
The score of query q for document d correlates to the
- * cosine-distance or dot-product between document and query vectors in a
+/**
+ * Expert: Scoring API.
+ *
+ *
Similarity defines the components of Lucene scoring.
+ * Overriding computation of these components is a convenient
+ * way to alter Lucene scoring.
+ *
+ *
Suggested reading:
+ *
+ * Introduction To Information Retrieval, Chapter 6.
+ *
+ *
The following describes how Lucene scoring evolves from
+ * underlying information retrieval models to (efficient) implementation.
+ *
+ *
Lucene combines
+ *
+ * Boolean model (BM) of Information Retrieval
+ * with
*
- * Vector Space Model (VSM) of Information Retrieval.
- * A document whose vector is closer to the query vector in that model is scored higher.
- *
- * The score is computed as follows:
- *
+ * Vector Space Model (VSM) of Information Retrieval -
+ * documents "approved" by BM are scored by VSM.
+ *
+ *
In VSM, documents and queries are represented as
+ * weighted vectors in a multi-dimensional space,
+ * where each distinct index term is a dimension,
+ * and weights are
+ * Tf-idf values.
+ *
+ *
It should be noted that VSM does not require weights to be Tf-idf values,
+ * but Tf-idf values are believed to produce search results of high quality,
+ * and so Lucene is using Tf-idf.
+ * The implementation of Tf and Idf in Lucene is discussed in more detail below,
+ * but for now, for completion, let's just say that
+ * Tf(t,x) is proportional to the number of occurrences of term t in x
+ * (x is either a document or a query) and
+ * idf(t) is proportional to the inverse of the
+ * number of index documents containing term t.
+ *
+ *
VSM score of document d for query q is the
+ *
+ * Cosine Similarity
+ * of the weighted query vectors V(q) and V(d):
+ *
+ *
+ *
+ *
+ * |
+ * cosine-similarity(q,d) =
+ * |
+ *
+ *
+ * | V(q) · V(d) |
+ * | ––––––––– |
+ * | |V(q)| |V(d)| |
+ *
+ * |
+ *
+ *
+ *
+ *
+ * Where V(q) · V(d) is the
+ * dot product
+ * of the weighted vectors,
+ * and |V(q)| and |V(d)| are their
+ * Euclidean norms.
+ *
+ * Note: the above equation can be viewed as the dot product of
+ * the normalized weighted vectors, in the sense that dividing
+ * V(q) by its euclidean norm is normalizing it to a unit vector.
+ *
+ *
This is, in principle, the scoring function of Lucene.
+ * However there are a few refinements, for both search quality and usability:
+ *
+ * - In order to prevent large documents from "taking over" just because
+ * a very large document may match all terms of the query, it is common
+ * practice to "punish" very large documents proportional to their length.
+ * For this, the score of each document is also multiplied by a document
+ * normalization doc-len-norm(d).
+ *
+ *
+ * - At indexing, users can specify that certain documents are more
+ * important than others, by assigning a document boost.
+ * For this, the score of each document is also multiplied by its boost value
+ * doc-boost(d).
+ *
+ *
+ * - Lucene is field based, hence each query term applies to a single
+ * field, document length normalization is by the length of the certain field,
+ * and document boost actually is a document field boost.
+ *
+ *
+ * - The same field can be added to a document during indexing several time,
+ * and so the boost of that field is the multiplication of the boosts of
+ * the separate additions (or parts) of that field within the document.
+ *
+ *
+ * - At search time users can specify boosts to each query, sub-query, and
+ * each query, hence the contribution of a query term to the score of
+ * a document is multiplied by the boost of that query term.
+ *
+ *
+ * - A document may match a multi term query without containing all
+ * the terms of that query (this is correct for some of the queries),
+ * and users can further reward documents matching more query terms
+ * through the coord-factor, which is usually larger when
+ * more terms are matched, and is multiplied by the score as well.
+ *
+ *
+ *
+ * Under the simplifying assumption of a single field in the index,
+ * here is the resulted scoring formula:
+ *
+ *
+ *
+ *
+ * |
+ * score(q,d) =
+ * coord-factor(q,d) ·
+ * query-boost(q) ·
+ * |
+ *
+ *
+ * | V(q) · V(d) |
+ * | ––––––––– |
+ * | |V(q)| |V(d)| |
+ *
+ * |
+ *
+ * · doc-len-norm(d)
+ * · doc-boost(d)
+ * |
+ *
+ *
+ *
+ *
+ * The above formula is a simplification in the sense that (1) terms and documents
+ * are fielded and (2) boosts are usually per query term rather than per query.
+ *
+ *
Lucene evidently omits some of these components from the scoring
+ * computation for performance considerations.
+ * In addition, some scoring components are computed in advance,
+ * and further aggregated in advance for an efficient score computation
+ * for each matching document:
+ *
+ *
+ * - Query-boost for the query (actually for each query term)
+ * is known when search starts.
+ *
+ *
+ * - Query Euclidean norm |V(q)| can be computed when search starts,
+ * as it is independent of the document being scored.
+ * From search optimization perscoective, it is a valid question
+ * why bother to normalize the query at all, because all
+ * scored documents will be multiplied by the same |V(q)|,
+ * and hence documents ranks (their order by score) will not
+ * be affected by this normalization.
+ * There are two good reasons to keep this normalization:
+ *
+ * - Recall that
+ *
+ * Cosine Similarity can be used find how similar
+ * two documents are. One can use Lucene for e.g.
+ * clustering, and use a document as a query to compute
+ * its similarity to other documents.
+ * In this use case it is important that the score of document d3
+ * for query d1 is comparable to the score of document d3
+ * for query d2. In other words, scores of a document for two
+ * distinct queries should be comparable.
+ * There are other applications that may require this.
+ * And this is exactly what normalizing the query vector V(q)
+ * provides: comparability (to a certain extent) of two or more queries.
+ *
+ *
+ * - Applying query normalization on the scores helps to keep the
+ * scores arround the unit vector, hence preventing loss of score data
+ * because of floating point precision limitations.
+ *
+ *
+ *
+ *
+ * - Document Euclidean norm |V(d)| is excluded by Lucene,
+ * for indexing performance considerstions (?).
+ *
+ *
+ * - Document length norm doc-len-norm(d) and document
+ * boost doc-bost(d) are known at indexing time.
+ * They are computed in advance and their multiplication
+ * is saved as a single value in the index: norm(d).
+ * (In the equations below, norm(t in d) means norm(field(t) in doc d)
+ * where field(t) is the field associated with term t.)
+ *
+ *
+ *
+ * Lucene's actual scoring function is derived from the above.
+ * The color codes demonstrate how actual scoring function components
+ * relate to those of the theoretical scoring function:
+ *
*
*
*
*
- *
+ *
* where
*
* -
@@ -79,9 +265,13 @@
* correlates to the term's frequency,
* defined as the number of times term t appears in the currently scored document d.
* Documents that have more occurrences of a given term receive a higher score.
+ * Note that tf(t in q) is assumed to be 1 for and there for it not appear in this equation,
+ * However if a qurey contains twice the same term, there will be
+ * two term-queries with that same term and hence the computation would still be correct (although
+ * not very efficient).
* The default computation for tf(t in d) in
* {@link org.apache.lucene.search.DefaultSimilarity#tf(float) DefaultSimilarity} is:
- *
+ *
*
*
*
*
- *
+ *
* -
*
* idf(t) stands for Inverse Document Frequency. This value
* correlates to the inverse of docFreq
* (the number of documents in which the term t appears).
* This means rarer terms give higher contribution to the total score.
+ * idf(t) appears for t in both the query and the document, and there it is squared in the equation.
* The default computation for idf(t) in
* {@link org.apache.lucene.search.DefaultSimilarity#idf(int, int) DefaultSimilarity} is:
- *
+ *
*
*
*
*
- *
+ *
* -
*
* coord(q,d)
@@ -140,7 +331,7 @@
* by the Similarity in effect at search time.
*
*
- *
+ *
* -
*
* queryNorm(q)
@@ -149,7 +340,7 @@
* This factor does not affect document ranking (since all ranked documents are multiplied by the same factor),
* but rather just attempts to make scores from different queries (or even different indexes) comparable.
* This is a search time factor computed by the Similarity in effect at search time.
- *
+ *
* The default computation in
* {@link org.apache.lucene.search.DefaultSimilarity#queryNorm(float) DefaultSimilarity}
* is:
@@ -173,12 +364,12 @@
*
*
*
- *
+ *
* The sum of squared weights (of the query terms) is
* computed by the query {@link org.apache.lucene.search.Weight} object.
* For example, a {@link org.apache.lucene.search.BooleanQuery boolean query}
* computes this value as:
- *
+ *
*
*
*
@@ -204,9 +395,9 @@
*
*
*
- *
+ *
*
- *
+ *
* -
*
* t.getBoost()
@@ -222,11 +413,11 @@
* {@link org.apache.lucene.search.Query#getBoost() getBoost()}.
*
*
- *
+ *
* -
*
* norm(t,d) encapsulates a few (indexing time) boost and length factors:
- *
+ *
*
* - Document boost - set by calling
* {@link org.apache.lucene.document.Document#setBoost(float) doc.setBoost()}
@@ -242,11 +433,11 @@
* LengthNorm is computed by the Similarity class in effect at indexing.
*
*
- *
+ *
*
* When a document is added to the index, all the above factors are multiplied.
* If the document has multiple fields with the same name, all their boosts are multiplied together:
- *
+ *
*
*