Solr
  1. Solr
  2. SOLR-5973

Pluggable Ranking Collectors and Merge Strategies

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9
    • Component/s: search
    • Labels:
      None

      Description

      This ticket introduces a new RankQuery and MergeStrategy to Solr. By extending the RankQuery class, and implementing it's interface, you can specify a custom ranking collector (TopDocsCollector) and distributed merge strategy for a Solr query.

      Sample syntax:

      q=hello&rq={!customRank param1=a param2=b}&wt=json&indent=true
      

      In the sample above the new "rq" (rank query) param:

      rq={!customRank  param1=a param2=b}

      points to a QParserPlugin that returns a Query that extends RankQuery. The RankQuery defines the custom ranking and merge strategy for the main query.

      The RankQuery impl will have to do several things:

      1) Implement the getTopDocsCollector() method to return a custom top docs ranking collector.
      2) Implement the wrap() method. The QueryComponent calls the wrap() method to wrap the RankQuery around the main query. This design allows the RankQuery to manage Query caching issues and implement custom Query explanations if needed.
      3) Implement hashCode() and equals() so the queryResultCache works properly with main query and custom ranking algorithm.
      4) Optionally implement a custom MergeStrategy to handle the merging of distributed results from the shards.

      1. SOLR-5973.patch
        7 kB
        Joel Bernstein
      2. SOLR-5973.patch
        72 kB
        Joel Bernstein
      3. SOLR-5973.patch
        72 kB
        Joel Bernstein
      4. SOLR-5973.patch
        72 kB
        Joel Bernstein
      5. SOLR-5973.patch
        69 kB
        Joel Bernstein
      6. SOLR-5973.patch
        67 kB
        Joel Bernstein
      7. SOLR-5973.patch
        66 kB
        Joel Bernstein
      8. SOLR-5973.patch
        63 kB
        Joel Bernstein
      9. SOLR-5973.patch
        55 kB
        Joel Bernstein
      10. SOLR-5973.patch
        55 kB
        Joel Bernstein
      11. SOLR-5973.patch
        45 kB
        Joel Bernstein
      12. SOLR-5973.patch
        43 kB
        Joel Bernstein
      13. SOLR-5973.patch
        41 kB
        Joel Bernstein
      14. SOLR-5973.patch
        38 kB
        Joel Bernstein
      15. SOLR-5973.patch
        7 kB
        Joel Bernstein
      16. SOLR-5973.patch
        5 kB
        Joel Bernstein

        Issue Links

          Activity

          Hide
          Joel Bernstein added a comment -

          Initial patch. Will deal with the custom distributed merge strategies in later patches.

          Show
          Joel Bernstein added a comment - Initial patch. Will deal with the custom distributed merge strategies in later patches.
          Hide
          Joel Bernstein added a comment -

          Added pluggable MergeStrategy to handle custom distributed merge logic.

          Show
          Joel Bernstein added a comment - Added pluggable MergeStrategy to handle custom distributed merge logic.
          Hide
          Otis Gospodnetic added a comment -

          Very exciting, Joel. Is there an example of how to do/use this somewhere?

          Show
          Otis Gospodnetic added a comment - Very exciting, Joel. Is there an example of how to do/use this somewhere?
          Hide
          Joel Bernstein added a comment -

          Thanks Otis. I'll be adding the test cases next week, which will show the pluggable collectors and merge strategy in action.

          Show
          Joel Bernstein added a comment - Thanks Otis. I'll be adding the test cases next week, which will show the pluggable collectors and merge strategy in action.
          Hide
          Joel Bernstein added a comment -

          Added tests for pluggable CollectorFactory via custom SearchComponent.

          Show
          Joel Bernstein added a comment - Added tests for pluggable CollectorFactory via custom SearchComponent.
          Hide
          Joel Bernstein added a comment -

          Added support for the QueryResultCache by adding the CollectorFactory to the QueryResultKey. This is needed because the CollectorFactory dictates the sort order. Also added support for QueryResultCache autoWarming with CollectorFactories.

          Show
          Joel Bernstein added a comment - Added support for the QueryResultCache by adding the CollectorFactory to the QueryResultKey. This is needed because the CollectorFactory dictates the sort order. Also added support for QueryResultCache autoWarming with CollectorFactories.
          Hide
          Joel Bernstein added a comment -

          Changed the QueryResultKeyTest to work with new QueryResultKey. Still needs test for QueryResultKeys with a non-null CollectorFactory.

          Show
          Joel Bernstein added a comment - Changed the QueryResultKeyTest to work with new QueryResultKey. Still needs test for QueryResultKeys with a non-null CollectorFactory.
          Hide
          Joel Bernstein added a comment -

          Added tests for QueryResultKey equality with CollectorFactories.

          Show
          Joel Bernstein added a comment - Added tests for QueryResultKey equality with CollectorFactories.
          Hide
          Joel Bernstein added a comment -

          Added MergeStrategyTest which ensures the basic plumbing for pluggable MergeStrategies is working.

          Show
          Joel Bernstein added a comment - Added MergeStrategyTest which ensures the basic plumbing for pluggable MergeStrategies is working.
          Hide
          Joel Bernstein added a comment -

          Patch with all unit tests passing.

          Show
          Joel Bernstein added a comment - Patch with all unit tests passing.
          Hide
          Joel Bernstein added a comment -

          Added mergeStrategy test using sort criteria other then score.

          Show
          Joel Bernstein added a comment - Added mergeStrategy test using sort criteria other then score.
          Hide
          Joel Bernstein added a comment -

          More mergeStrategy tests. Passes precommit.

          Show
          Joel Bernstein added a comment - More mergeStrategy tests. Passes precommit.
          Hide
          Joel Bernstein added a comment -

          Added getCost() method to MergeStrategies.

          Show
          Joel Bernstein added a comment - Added getCost() method to MergeStrategies.
          Hide
          Joel Bernstein added a comment -

          Added MergeStrategy ordering test.

          Show
          Joel Bernstein added a comment - Added MergeStrategy ordering test.
          Hide
          Joel Bernstein added a comment -

          New design where TopDocCollector and MergeStrategy can be injected via QParserPlugin.

          Show
          Joel Bernstein added a comment - New design where TopDocCollector and MergeStrategy can be injected via QParserPlugin.
          Hide
          Otis Gospodnetic added a comment -

          Joel Bernstein how does this play with SOLR-1680? I suspect it doesn't at all, right? Do you think SOLR-1680 has any merit or should it be Won't Fix-ed?

          Show
          Otis Gospodnetic added a comment - Joel Bernstein how does this play with SOLR-1680 ? I suspect it doesn't at all, right? Do you think SOLR-1680 has any merit or should it be Won't Fix-ed?
          Hide
          Joel Bernstein added a comment -

          SOLR-1680 is covered by this ticket and by the CollapsingQParserPlugin. So I think we can shutdown SOLR-1680 as Won't fix and point to this ticket.

          Show
          Joel Bernstein added a comment - SOLR-1680 is covered by this ticket and by the CollapsingQParserPlugin. So I think we can shutdown SOLR-1680 as Won't fix and point to this ticket.
          Hide
          ASF subversion and git services added a comment -

          Commit 1594698 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1594698 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1594698 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1594698 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies
          Hide
          ASF subversion and git services added a comment -

          Commit 1594749 from Joel Bernstein in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1594749 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1594749 from Joel Bernstein in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1594749 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies
          Hide
          Joel Bernstein added a comment -

          A new patch. Solves issues that were found while working on SOLR-6088.

          Show
          Joel Bernstein added a comment - A new patch. Solves issues that were found while working on SOLR-6088 .
          Hide
          ASF subversion and git services added a comment -

          Commit 1597775 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1597775 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1597775 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1597775 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies
          Hide
          ASF subversion and git services added a comment -

          Commit 1597796 from Joel Bernstein in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1597796 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1597796 from Joel Bernstein in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1597796 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies
          Hide
          ASF subversion and git services added a comment -

          Commit 1597921 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1597921 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1597921 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1597921 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies
          Hide
          ASF subversion and git services added a comment -

          Commit 1597923 from Joel Bernstein in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1597923 ]

          SOLR-5973: Pluggable Ranking Collectors and Merge Strategies

          Show
          ASF subversion and git services added a comment - Commit 1597923 from Joel Bernstein in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1597923 ] SOLR-5973 : Pluggable Ranking Collectors and Merge Strategies

            People

            • Assignee:
              Joel Bernstein
              Reporter:
              Joel Bernstein
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development