Lucene - Core
  1. Lucene - Core
  2. LUCENE-3029

MultiPhraseQuery assigns different scores to identical docs when using 0 pos-incr

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.4, 3.2, 4.0-ALPHA
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      If you have two identical docs with tokens a b c all zero pos-incr (ie
      they occur on the same position), and you run a MultiPhraseQuery with
      [a, b] and [c] (all pos incr 0)... then the two docs will get
      different scores despite being identical.

      Admittedly it's a strange query... but I think the scorer ought to
      count the phrase as having tf=1 for each doc.

      The problem is that we are missing a tie-breaker for the PhraseQuery
      used by ExactPhraseScorer, and so the PQ ends up flip/flopping such
      that every other document gets the same score. Ie, even docIDs all
      get one score and odd docIDs all get another score.

      Once I added the hard tie-breaker (ord) the scores are the same.

      However... there's a separate bug, that can over-count the tf, such
      that if I create the MPQ like this:

        mpq.add(new Term[] {new Term("field", "a")}, 0);
        mpq.add(new Term[] {new Term("field", "b"), new Term("field", "c")}, 0);
      

      I get tf=2 per doc, but if I create it like this:

        mpq.add(new Term[] {new Term("field", "b"), new Term("field", "c")}, 0);
        mpq.add(new Term[] {new Term("field", "a")}, 0);
      

      I get tf=1 (which I think is correct?).

      This happens because MultipleTermPositions freely returns the same
      position more than once: it just unions the positions of the two
      streams, so when both have their term at pos=0, you'll get pos=0
      twice, which is not good and leads to over-counting tf.

      Unfortunately, I don't see a performant way to fix that... and I'm not
      sure that it really matters that much in practice.

      1. LUCENE-3029.patch
        10 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch.

        Show
        Michael McCandless added a comment - Patch.
        Hide
        Robert Muir added a comment -

        Bulk closing for 3.2

        Show
        Robert Muir added a comment - Bulk closing for 3.2

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development