Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-6687

MLT term frequency calculation bug



    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 5.2.1, 6.0
    • 5.2.2, 8.1, 9.0
    • None
    • OS X v10.10.4; Solr 5.2.1

    • New, Patch Available


      In org.apache.lucene.queries.mlt.MoreLikeThis, there's a method retrieveTerms that receives a Map of fields, i.e. a document basically, but it doesn't have to be an existing doc.

      There are 2 for loops, one inside the other, which both loop through the same set of fields.
      That effectively doubles the term frequency for all the terms from fields that we provide in MLT QP qf parameter.
      It basically goes two times over the list of fields and accumulates the term frequencies from all fields into termFreqMap.

      The private method retrieveTerms is only called from one public method, the version of overloaded method like that receives a Map: so that private class member fieldNames is always derived from retrieveTerms's argument fields.

      Uh, I don't understand what I wrote myself, but that basically means that, by the time retrieveTerms method gets called, its parameter fields and private member fieldNames always contain the same list of fields.

      Here's the proof:
      These are the final results of the calculation:

      And this is the actual thread_id:TID0009 document, where those values were derived from (from fields title_mlt and pagetext_mlt):

      Now, let's further test this hypothesis by seeing MLT QP in action from the AdminUI.
      Let's try to find docs that are More Like doc TID0009.
      Here's the interesting part, the query:

      q={!mlt qf=pagetext_mlt,title_mlt mintf=14 mindf=2 minwl=3 maxwl=15}TID0009

      We just saw, in the last image above, that the term accumulator appears 7 times in TID0009 doc, but the accumulator's TF was calculated as 14.
      By using mintf=14, we say that, when calculating similarity, we don't want to consider terms that appear less than 14 times (when terms from fields title_mlt and pagetext_mlt are merged together) in TID0009.
      I added the term accumulator in only one other document (TID0004), where it appears only once, in the field title_mlt.

      Let's see what happens when we use mintf=15:

      I should probably mention that multiple fields (qf) work because I applied the patch: SOLR-7143.

      Bug, no?


        1. buggy-method-usage.png
          381 kB
          Marko Bonaci
        2. LUCENE-6687.patch
          14 kB
          Alessandro Benedetti
        3. LUCENE-6687.patch
          14 kB
          Alessandro Benedetti
        4. LUCENE-6687.patch
          3 kB
          Alessandro Benedetti
        5. LUCENE-6687.patch
          2 kB
          Marko Bonaci
        6. solr-mlt-tf-doubling-bug.png
          414 kB
          Marko Bonaci
        7. solr-mlt-tf-doubling-bug-results.png
          272 kB
          Marko Bonaci
        8. solr-mlt-tf-doubling-bug-verify-accumulator-mintf14.png
          498 kB
          Marko Bonaci
        9. solr-mlt-tf-doubling-bug-verify-accumulator-mintf15.png
          336 kB
          Marko Bonaci
        10. terms-accumulator.png
          101 kB
          Marko Bonaci
        11. terms-angry.png
          99 kB
          Marko Bonaci
        12. terms-glass.png
          97 kB
          Marko Bonaci
        13. terms-how.png
          97 kB
          Marko Bonaci



            teofili Tommaso Teofili
            mbonaci Marko Bonaci
            2 Vote for this issue
            7 Start watching this issue



              Time Tracking

                Original Estimate - Not Specified
                Not Specified
                Remaining Estimate - 0h
                Time Spent - 1.5h