Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.1, 6.0
    • Component/s: core/query/scoring
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      CustomScoreQuery wrongly applies boost^2 instead of boost.

      It wrongly incorporates its boost into the normalization factor passed down to subquery (like booleanquery does) and also multiplies it directly in its scorer.

      The only reason the test passes today is because it compares raw score magnitudes when querynorm is on, which normalizes this away.

      Changing the test to use newSearcher() demonstrates the brokenness.

      1. LUCENE-4935.patch
        5 kB
        Robert Muir
      2. LUCENE-4935.patch
        3 kB
        Robert Muir

        Activity

        Hide
        Uwe Schindler added a comment -

        Oh oh! Thanks for investigating this! Is this new in 4.x? I used CustomScoreQuery in the past in 3.6 and did not recognize that bug.

        Show
        Uwe Schindler added a comment - Oh oh! Thanks for investigating this! Is this new in 4.x? I used CustomScoreQuery in the past in 3.6 and did not recognize that bug.
        Hide
        Robert Muir added a comment -

        Uwe: yes the bug is in 3.6 too: see CustomWeight's normalize (boost1)

            @Override
            public void normalize(float norm) {
              norm *= getBoost(); // incorporate boost
              subQueryWeight.normalize(norm);
        

        and CustomScorer.score (boost2)

              return qWeight * provider.customScore(subQueryScorer.docID(), subQueryScorer.score(), vScores);
        
        Show
        Robert Muir added a comment - Uwe: yes the bug is in 3.6 too: see CustomWeight's normalize (boost1) @Override public void normalize( float norm) { norm *= getBoost(); // incorporate boost subQueryWeight.normalize(norm); and CustomScorer.score (boost2) return qWeight * provider.customScore(subQueryScorer.docID(), subQueryScorer.score(), vScores);
        Hide
        Robert Muir added a comment -

        Here is an updated patch thats more correct in my opinion (but the first one fixes the bug).

        In my opinion this query should not 'distribute' any boosts down to subQuery, because an arbitrary function will be executed on that score that might not adhere to the distributive law.

        So if someone has a BooleanQuery(CustomScoreQuery(TermQuery)))^5, but their CustomScoreProvider scores with log(subQueryScore) + valSrcScore, they might wonder what happened to their boost of 5 on the outer booleanquery.

        This is more of a corner case, but i think it simplifies the thing so it acts more like ConstantScoreQuery.

        Show
        Robert Muir added a comment - Here is an updated patch thats more correct in my opinion (but the first one fixes the bug). In my opinion this query should not 'distribute' any boosts down to subQuery, because an arbitrary function will be executed on that score that might not adhere to the distributive law. So if someone has a BooleanQuery(CustomScoreQuery(TermQuery)))^5, but their CustomScoreProvider scores with log(subQueryScore) + valSrcScore, they might wonder what happened to their boost of 5 on the outer booleanquery. This is more of a corner case, but i think it simplifies the thing so it acts more like ConstantScoreQuery.
        Hide
        Michael McCandless added a comment -

        +1 for the 2nd approach.

        Show
        Michael McCandless added a comment - +1 for the 2nd approach.
        Hide
        Steve Rowe added a comment -

        If there are no objections, I'd like to backport this to 4.3.1.

        Show
        Steve Rowe added a comment - If there are no objections, I'd like to backport this to 4.3.1.
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk closing after 4.3.1 release

        Show
        Shalin Shekhar Mangar added a comment - Bulk closing after 4.3.1 release

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development