diff --git lucene/src/java/org/apache/lucene/search/package.html lucene/src/java/org/apache/lucene/search/package.html index 082b5cb..d1a79c3 100644 --- lucene/src/java/org/apache/lucene/search/package.html +++ lucene/src/java/org/apache/lucene/search/package.html @@ -202,61 +202,16 @@ org.apache.lucene.search.IndexSearcher#search(Query,Filter,int)}. Levenshtein (edit) distance. This type of query can be useful when accounting for spelling variations in the collection.

- -

Changing Similarity

-

Chances are DefaultSimilarity is sufficient for all - your searching needs. - However, in some applications it may be necessary to customize your Similarity implementation. For instance, some - applications do not need to - distinguish between shorter and longer documents (see a "fair" similarity).

-

To change Similarity, one must do so for both indexing and - searching, and the changes must happen before - either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it - just isn't well-defined what is going to happen. -

+

Ranking Models

+ +

The information retrieval models used in ranking can be found in the +{@linkplain org.apache.lucene.search.similarities similarities} package. Refer +to the package documentation to see how to use or change +{@linkplain org.apache.lucene.search.similarities.Similarity Similarities}.

-

To make this change, implement your own Similarity (likely - you'll want to simply subclass - DefaultSimilarity) and then use the new - class by calling - IndexWriter.setSimilarity - before indexing and - Searcher.setSimilarity - before searching. -

-

- If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at Overriding Similarity. - In summary, here are a few use cases: -

    -

  1. SweetSpotSimilarity — SweetSpotSimilarity gives small increases - as the frequency increases a small amount - and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is - more significant.

    2. -

  2. Overriding tf — In some applications, it doesn't matter what the score of a document is as long as a - matching term occurs. In these - cases people have overridden Similarity to return 1 from the tf() method.

    3. -

  3. Changing Length Normalization — By overriding lengthNorm, - it is possible to discount how the length of a field contributes - to a score. In DefaultSimilarity, - lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be - 1 / (numTerms in field), all fields will be treated - "fairly".

    4. -
- In general, Chris Hostetter sums it up best in saying (from the Lucene users's mailing list): -
[One would override the Similarity in] ... any situation where you know more about your data then just - that - it's "text" is a situation where it *might* make sense to to override your - Similarity method.
-

Changing Scoring — Expert Level

diff --git lucene/src/java/org/apache/lucene/search/similarities/BM25Similarity.java lucene/src/java/org/apache/lucene/search/similarities/BM25Similarity.java index c6eb435..4850c87 100644 --- lucene/src/java/org/apache/lucene/search/similarities/BM25Similarity.java +++ lucene/src/java/org/apache/lucene/search/similarities/BM25Similarity.java @@ -28,7 +28,10 @@ import org.apache.lucene.util.SmallFloat; import org.apache.lucene.util.TermContext; /** - * BM25 Similarity. + * BM25 Similarity. Introduced in Stephen E. Robertson, Steve Walker, + * Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at TREC-3. + * In Proceedings of the Third Text REtrieval Conference (TREC 1994). + * Gaithersburg, USA, November 1994. */ public class BM25Similarity extends Similarity { private final float k1; diff --git lucene/src/java/org/apache/lucene/search/similarities/EasySimilarity.java lucene/src/java/org/apache/lucene/search/similarities/EasySimilarity.java index 870f3fb..6cc13ea 100644 --- lucene/src/java/org/apache/lucene/search/similarities/EasySimilarity.java +++ lucene/src/java/org/apache/lucene/search/similarities/EasySimilarity.java @@ -31,7 +31,12 @@ import org.apache.lucene.util.TermContext; /** * A subclass of {@code Similarity} that provides a simplified API for its - * descendants. + * descendants. Subclasses are only required to implement the {@link #score} + * and {@link #toString()} methods. Implementing + * {@link #explain(Explanation, EasyStats, int, float, float)} is optional, + * inasmuch as EasySimilarity already provides a basic explanation of the score + * and the term frequency. However, implementers of a subclass are encouraged to + * include as much detail about the scoring method as possible. * @lucene.experimental */ public abstract class EasySimilarity extends Similarity { @@ -138,10 +143,10 @@ public abstract class EasySimilarity extends Similarity { * Explains the score. The implementation here provides a basic explanation * in the format score(name-of-similarity, doc=doc-id, * freq=term-frequency), computed from:, and - * attaches the score (computed via the {@link #score(EasyStats, float, byte)} + * attaches the score (computed via the {@link #score(EasyStats, float, float)} * method) and the explanation for the term frequency. Subclasses content with * this format may add additional details in - * {@link #explain(Explanation, EasyStats, int, float, byte)}. + * {@link #explain(Explanation, EasyStats, int, float, float)}. * * @param stats the corpus level statistics. * @param doc the document id. diff --git lucene/src/java/org/apache/lucene/search/similarities/package.html lucene/src/java/org/apache/lucene/search/similarities/package.html new file mode 100644 index 0000000..7a05aa5 --- /dev/null +++ lucene/src/java/org/apache/lucene/search/similarities/package.html @@ -0,0 +1,162 @@ + + + + + + + +This package contains the various ranking models that can be used in Lucene. The +abstract class {@link org.apache.lucene.search.similarities.Similarity} serves +as the base for ranking functions. For searching, users can employ the models +already implemented or create their own by extending one of the classes in this +package. + +

Table Of Contents

+

+

    +
  1. Summary of the Ranking Methods
    2. +
  2. Similarity Providers
    3. +
  3. Changing the Similarity
    4. +
+

+ + + +

Summary of the Ranking Methods

+ +

{@link org.apache.lucene.search.similarities.DefaultSimilarity} is the original Lucene +scoring function. It is based on a highly optimized Vector Space Model. For more +information, see {@link org.apache.lucene.search.similarities.TFIDFSimilarity}.

+ +

{@link org.apache.lucene.search.similarities.BM25Similarity} is an optimized +implementation of the successful Okapi BM25 model.

+ +

{@link org.apache.lucene.search.similarities.EasySimilarity} provides a basic +implementation of the Similarity contract and exposes a highly simplified +interface, which makes it an ideal starting point for new ranking functions. +Lucene ships the following EasySimilarity-based methods: + + +

+ +Since EasySimilarity is not optimized to the same extent as +{@link org.apache.lucene.search.similarities.DefaultSimilarity} and +{@link org.apache.lucene.search.similarities.BM25Similarity}, a difference in +performance is to be expected when using the methods listed above. However, +optimizations can always be implemented in subclasses; see +below.

+ + + +

Similarity Providers

+ +

{@link org.apache.lucene.search.similarities.SimilarityProvider}s instantiate +Similarities and compute the normalization value for the query. +{@link org.apache.lucene.search.similarities.DefaultSimilarityProvider} is used +by both {@link org.apache.lucene.search.similarities.DefaultSimilarity} and +(indirectly) by {@link org.apache.lucene.search.similarities.BM25Similarity}, +and the query norm is defined as 1 / (numTerms in field)^0.5.

+ + + +

Changing Similarity

+ +

Chances are the available Similarities are sufficient for all + your searching needs. + However, in some applications it may be necessary to customize your Similarity implementation. For instance, some + applications do not need to + distinguish between shorter and longer documents (see a "fair" similarity).

+ +

To change {@link org.apache.lucene.search.similarities.Similarity}, one must do so for both indexing and + searching, and the changes must happen before + either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it + just isn't well-defined what is going to happen. +

+ +

To make this change, implement your own {@link org.apache.lucene.search.similarities.Similarity} (likely + you'll want to simply subclass an existing method, be it + {@link org.apache.lucene.search.similarities.DefaultSimilarity} or a descendant of + {@link org.apache.lucene.search.similarities.EasySimilarity}) and + {@link org.apache.lucene.search.similarities.SimilarityProvider}, and then + use the new class by calling + {@link org.apache.lucene.index.IndexWriterConfig#setSimilarityProvider(SimilarityProvider)} + before indexing and + {@link org.apache.lucene.search.IndexSearcher#setSimilarityProvider(SimilarityProvider)} + before searching. +

+ +

Extending {@linkplain org.apache.lucene.search.similarities.EasySimilarity}

+

+The easiest way to quickly implement a new ranking method is to extend +{@link org.apache.lucene.search.similarities.EasySimilarity}, which provides +basic implementations for the low level . Subclasses are only required to +implement the {@link org.apache.lucene.search.similarities.EasySimilarity#score(EasyStats, float, float)} +and {@link org.apache.lucene.search.similarities.EasySimilarity#toString()} +methods.

+ +

Another options is to extend one of the frameworks +based on {@link org.apache.lucene.search.similarities.EasySimilarity}. These +Similarities are implemented modularly, e.g. +{@link org.apache.lucene.search.similarities.DFRSimilarity} delegates +computation of the three parts of its formula to the classes +{@link org.apache.lucene.search.similarities.BasicModel}, +{@link org.apache.lucene.search.similarities.AfterEffect} and +{@link org.apache.lucene.search.similarities.Normalization}. Instead of +subclassing the Similarity, one can simply introduce a new basic model and tell +{@link org.apache.lucene.search.similarities.DFRSimilarity} to use it.

+ +

Changing {@linkplain org.apache.lucene.search.similarities.DefaultSimilarity}

+

+ If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at Overriding Similarity. + In summary, here are a few use cases: +

    +

  1. The SweetSpotSimilarity in + org.apache.lucene.misc gives small + increases as the frequency increases a small amount + and then greater increases when you hit the "sweet spot", i.e. where + you think the frequency of terms is more significant.

    2. +

  2. Overriding tf — In some applications, it doesn't matter what the score of a document is as long as a + matching term occurs. In these + cases people have overridden Similarity to return 1 from the tf() method.

    3. +

  3. Changing Length Normalization — By overriding + {@link org.apache.lucene.search.similarities.Similarity#computeNorm(FieldInvertState state)}, + it is possible to discount how the length of a field contributes + to a score. In {@link org.apache.lucene.search.similarities.DefaultSimilarity}, + lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be + 1 / (numTerms in field), all fields will be treated + "fairly".

    4. +
+ In general, Chris Hostetter sums it up best in saying (from the Lucene users's mailing list): +
[One would override the Similarity in] ... any situation where you know more about your data then just + that + it's "text" is a situation where it *might* make sense to to override your + Similarity method.
+

+ + +