Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3
    • Component/s: search
    • Labels:
      None

      Description

      Searching over multiple shards and aggregating results.
      Motivated by http://wiki.apache.org/solr/DistributedSearch

      1. solr-dist-faceting-non-ascii-all.patch
        6 kB
        Lars Kotthoff
      2. shards.start_rows.patch
        3 kB
        Brian Whitman
      3. shards_qt.patch
        0.8 kB
        Yonik Seeley
      4. fedsearch.stu.patch
        92 kB
        Stu Hood
      5. fedsearch.stu.patch
        94 kB
        Stu Hood
      6. fedsearch.patch
        57 kB
        Sharad Agarwal
      7. fedsearch.patch
        72 kB
        Sharad Agarwal
      8. fedsearch.patch
        86 kB
        Sharad Agarwal
      9. fedsearch.patch
        162 kB
        Sharad Agarwal
      10. fedsearch.patch
        109 kB
        Sabyasachi Dalal
      11. fedsearch.patch
        135 kB
        Sabyasachi Dalal
      12. fedsearch.patch
        135 kB
        Sabyasachi Dalal
      13. distributed.patch
        122 kB
        Yonik Seeley
      14. distributed.patch
        123 kB
        Yonik Seeley
      15. distributed.patch
        135 kB
        Yonik Seeley
      16. distributed.patch
        83 kB
        Yonik Seeley
      17. distributed.patch
        87 kB
        Yonik Seeley
      18. distributed.patch
        87 kB
        Yonik Seeley
      19. distributed.patch
        98 kB
        Yonik Seeley
      20. distributed.patch
        100 kB
        Yonik Seeley
      21. distributed.patch
        113 kB
        Yonik Seeley
      22. distributed.patch
        113 kB
        Yonik Seeley
      23. distributed.patch
        114 kB
        Yonik Seeley
      24. distributed.patch
        129 kB
        Yonik Seeley
      25. distributed_pjaol.patch
        93 kB
        patrick o'leary
      26. distributed_facet_count_bugfix.patch
        1 kB
        Jayson Minard
      27. distributed_add_tests_for_intended_behavior.patch
        3 kB
        Jayson Minard

        Issue Links

          Activity

          Uwe Schindler made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends upon SOLR-443 [ SOLR-443 ]
          Gavin made changes -
          Link This issue depends on SOLR-443 [ SOLR-443 ]
          Grant Ingersoll made changes -
          Link This issue relates to SOLR-788 [ SOLR-788 ]
          Yonik Seeley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Brian Whitman made changes -
          Attachment shards.start_rows.patch [ 12385455 ]
          Lars Kotthoff made changes -
          Link This issue depends on SOLR-443 [ SOLR-443 ]
          Lars Kotthoff made changes -
          Link This issue is related to SOLR-592 [ SOLR-592 ]
          Hoss Man made changes -
          Fix Version/s 1.3 [ 12312486 ]
          Otis Gospodnetic made changes -
          Comment [ Hi / Hallo,

          Thanks for your mail. Unfortunately, I won't be able to answer it
          soon. I'm on vacation till June 2nd without access to my mails.

          ~~~~

          Vielen Dank für die Email. Leider werde ich nicht sofort antworten.
          Ich bin bis 2. Juni im Urlaub ohne Zugriff auf mein Postfach.

          -Gunnar

          --
          Gunnar Wagenknecht
          gunnar@wagenknecht.org
          http://wagenknecht.org/
          ]
          Lars Kotthoff made changes -
          Attachment solr-dist-faceting-non-ascii-all.patch [ 12382364 ]
          Yonik Seeley made changes -
          Attachment shards_qt.patch [ 12381157 ]
          Jayson Minard made changes -
          Link This issue is related to SOLR-507 [ SOLR-507 ]
          Jayson Minard made changes -
          Jayson Minard made changes -
          Attachment distributed_facet_count_bugfix.patch [ 12378136 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12376452 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12375797 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12375793 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12375791 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12374735 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12374298 ]
          patrick o'leary made changes -
          Attachment distributed_pjaol.patch [ 12373211 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372957 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372875 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372867 ]
          patrick o'leary made changes -
          Attachment distributed_trunk.patch [ 12372727 ]
          Sean Timm made changes -
          Comment [ I'm receiving both patch errors and compile errors from Yonik's latest patch (03/Jan/08) against head on the trunk (r. 610010). I ignore the errors on the two Test files. It fails to remove the handler/SearchHandler.java
          {noformat}
          % patch -p0 -u < ~/distributed.patch
          [...]
          patching file src/java/org/apache/solr/handler/SearchHandler.java
          Reversed (or previously applied) patch detected! Assume -R? [n]
          Apply anyway? [n] y
          Hunk #1 FAILED at 1.
          File src/java/org/apache/solr/handler/SearchHandler.java is not empty after patch, as expected
          1 out of 1 hunk FAILED -- saving rejects to file src/java/org/apache/solr/handler/SearchHandler.java.rej
          {noformat}

          After removing handler/SearchHandler.java, the build errors that I am getting are:
          {noformat}
          % ant compile
          Buildfile: build.xml

          init-forrest-entities:

          compile-common:

          compile:
              [javac] Compiling 84 source files to /home/timmsc/svn.apache.org/lucene/solr/trunk/build/core
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:303: 'class' or 'interface' expected
              [javac] package org.apache.solr.handler.component;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:305: 'class' or 'interface' expected
              [javac] import org.apache.solr.handler.RequestHandlerBase;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:306: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.util.NamedList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:307: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.util.RTimer;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:308: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.util.SimpleOrderedMap;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:309: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.params.CommonParams;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:310: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.params.ModifiableSolrParams;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:311: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.SolrException;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:312: 'class' or 'interface' expected
              [javac] import org.apache.solr.request.SolrQueryRequest;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:313: 'class' or 'interface' expected
              [javac] import org.apache.solr.request.SolrQueryResponse;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:314: 'class' or 'interface' expected
              [javac] import org.apache.solr.client.solrj.SolrServer;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:315: 'class' or 'interface' expected
              [javac] import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:316: 'class' or 'interface' expected
              [javac] import org.apache.solr.util.plugin.SolrCoreAware;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:317: 'class' or 'interface' expected
              [javac] import org.apache.solr.core.SolrCore;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:318: 'class' or 'interface' expected
              [javac] import org.apache.lucene.queryParser.ParseException;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:320: 'class' or 'interface' expected
              [javac] import java.util.logging.Logger;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:321: 'class' or 'interface' expected
              [javac] import java.util.Collection;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:322: 'class' or 'interface' expected
              [javac] import java.util.List;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:323: 'class' or 'interface' expected
              [javac] import java.util.ArrayList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java:324: 'class' or 'interface' expected
              [javac] import java.util.LinkedList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:273: 'class' or 'interface' expected
              [javac] package org.apache.solr.handler.component;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:275: 'class' or 'interface' expected
              [javac] import org.apache.lucene.search.SortComparatorSource;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:276: 'class' or 'interface' expected
              [javac] import org.apache.lucene.search.SortField;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:277: 'class' or 'interface' expected
              [javac] import org.apache.lucene.util.PriorityQueue;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:278: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.util.NamedList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:279: 'class' or 'interface' expected
              [javac] import org.apache.solr.search.MissingStringLastComparatorSource;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:281: 'class' or 'interface' expected
              [javac] import java.text.Collator;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:282: 'class' or 'interface' expected
              [javac] import java.util.Comparator;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:283: 'class' or 'interface' expected
              [javac] import java.util.Locale;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:284: 'class' or 'interface' expected
              [javac] import java.util.List;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardDoc.java:285: 'class' or 'interface' expected
              [javac] import java.util.ArrayList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardRequest.java:60: 'class' or 'interface' expected
              [javac] package org.apache.solr.handler.component;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardRequest.java:62: 'class' or 'interface' expected
              [javac] import org.apache.solr.client.solrj.response.QueryResponse;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardRequest.java:63: 'class' or 'interface' expected
              [javac] import org.apache.solr.common.params.ModifiableSolrParams;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardRequest.java:65: 'class' or 'interface' expected
              [javac] import java.util.ArrayList;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/ShardRequest.java:66: 'class' or 'interface' expected
              [javac] import java.util.List;
              [javac] ^
              [javac] 36 errors

          BUILD FAILED
          /home/timmsc/svn.apache.org/lucene/solr/trunk/build.xml:224: The following error occurred while executing this line:
          /home/timmsc/svn.apache.org/lucene/solr/trunk/build.xml:110: Compile failed; see the compiler error output for details.

          Total time: 1 second
          {noformat}

          A good number of the errors are because 3 of the files are duplicated inline after the patch. After fixing this I still get 17 errors.
          {noformat}
          % ant compile
          Buildfile: build.xml

          init-forrest-entities:

          compile-common:

          compile:
              [javac] Compiling 84 source files to /home/timmsc/svn.apache.org/lucene/solr/trunk/build/core
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxDebugComponent.java:45: cannot find symbol
              [javac] symbol : constructor SearchComponent(org.apache.solr.handler.component.SearchHandler)
              [javac] location: class org.apache.solr.handler.component.SearchComponent
              [javac] super(handler);
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxDebugComponent.java:51: method does not override a method from its superclass
              [javac] @Override
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxQueryComponent.java:57: cannot find symbol
              [javac] symbol : constructor QueryComponent(org.apache.solr.handler.component.SearchHandler)
              [javac] location: class org.apache.solr.handler.component.QueryComponent
              [javac] super(handler);
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxQueryComponent.java:63: method does not override a method from its superclass
              [javac] @Override
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxQueryComponent.java:204: cannot find symbol
              [javac] symbol : method getSortSpec(org.apache.solr.request.SolrQueryRequest)
              [javac] location: class org.apache.solr.util.SolrPluginUtils
              [javac] builder.setSortSpec(SolrPluginUtils.getSortSpec(req) );
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/component/DisMaxResponseBuilder.java:37: cannot find symbol
              [javac] symbol : constructor ResponseBuilder(org.apache.solr.request.SolrQueryRequest)
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] super(req);
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/FedSearchComponent.java:64: cannot find symbol
              [javac] symbol : constructor SearchComponent(org.apache.solr.handler.component.SearchHandler)
              [javac] location: class org.apache.solr.handler.component.SearchComponent
              [javac] super(handler);
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/FedSearchComponent.java:91: cannot find symbol
              [javac] symbol : variable handler
              [javac] location: class org.apache.solr.handler.federated.component.FedSearchComponent
              [javac] SolrCore.getSolrCore().execute(handler, localReq, response);
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/AuxiliaryQPhaseComponent.java:83: cannot find symbol
              [javac] symbol : variable request_HL_and_MLT_Info_InMainPhase
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] rspBuilder.request_HL_and_MLT_Info_InMainPhase = false;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/AuxiliaryQPhaseComponent.java:84: cannot find symbol
              [javac] symbol : variable request_fields_InMainPhase
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] rspBuilder.request_fields_InMainPhase = false;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/AuxiliaryQPhaseComponent.java:88: cannot find symbol
              [javac] symbol : method skipProcess(org.apache.solr.request.SolrQueryRequest,org.apache.solr.request.SolrQueryResponse)
              [javac] location: class org.apache.solr.handler.federated.component.FedSearchComponent
              [javac] if(super.skipProcess(req, rsp) ||
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/GlobalCollectionStatComponent.java:138: cannot find symbol
              [javac] symbol : method skipProcess(org.apache.solr.request.SolrQueryRequest,org.apache.solr.request.SolrQueryResponse)
              [javac] location: class org.apache.solr.handler.federated.component.FedSearchComponent
              [javac] if(super.skipProcess(req, rsp) ||
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/GlobalCollectionStatComponent.java:195: cannot find symbol
              [javac] symbol : variable extractedTerms
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] rspBuilder.extractedTerms = extractedTerms;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/GlobalCollectionStatComponent.java:205: cannot find symbol
              [javac] symbol : variable extractedTerms
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] final Set<String> extractedTerms = rspBuilder.extractedTerms;
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/MainQPhaseComponent.java:74: cannot find symbol
              [javac] symbol : method skipProcess(org.apache.solr.request.SolrQueryRequest,org.apache.solr.request.SolrQueryResponse)
              [javac] location: class org.apache.solr.handler.federated.component.FedSearchComponent
              [javac] if(super.skipProcess(req, rsp) ||
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/MainQPhaseComponent.java:208: cannot find symbol
              [javac] symbol : variable request_fields_InMainPhase
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] if(rspBuilder.request_fields_InMainPhase){
              [javac] ^
              [javac] /home/timmsc/svn.apache.org/lucene/solr/trunk/src/java/org/apache/solr/handler/federated/component/MainQPhaseComponent.java:238: cannot find symbol
              [javac] symbol : variable request_HL_and_MLT_Info_InMainPhase
              [javac] location: class org.apache.solr.handler.component.ResponseBuilder
              [javac] rspBuilder.request_HL_and_MLT_Info_InMainPhase
              [javac] ^
              [javac] Note: Some input files use or override a deprecated API.
              [javac] Note: Recompile with -Xlint:deprecation for details.
              [javac] Note: Some input files use unchecked or unsafe operations.
              [javac] Note: Recompile with -Xlint:unchecked for details.
              [javac] 17 errors

          BUILD FAILED
          /home/timmsc/svn.apache.org/lucene/solr/trunk/build.xml:224: The following error occurred while executing this line:
          /home/timmsc/svn.apache.org/lucene/solr/trunk/build.xml:110: Compile failed; see the compiler error output for details.

          Total time: 2 seconds
          {noformat} ]
          patrick o'leary made changes -
          Attachment distributed_trunk.patch [ 12372727 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372472 ]
          Ryan McKinley made changes -
          Link This issue incorporates SOLR-446 [ SOLR-446 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372179 ]
          Yonik Seeley made changes -
          Attachment distributed.patch [ 12372134 ]
          Sabyasachi Dalal made changes -
          Attachment fedsearch.patch [ 12371581 ]
          Sabyasachi Dalal made changes -
          Attachment fedsearch.patch [ 12371501 ]
          Sabyasachi Dalal made changes -
          Attachment fedsearch.patch [ 12371498 ]
          Sabyasachi Dalal made changes -
          Comment [ I have fixed and updated the patch with trunk version 600419. It is integrated with the re-opened SOLR-281 patch.
          I have added the configuration for the three distributed-search components in the solrconfig.xml, under "/search" request handler. So, the distributed search works with /search request only.

          Couple of issues :
          1. The dist search components need the reference to the SearchHandler. So for now , i have hard coded the "/search" pattern in the FedSearchComponent.
          2. Need a clean way to load common init params for the dist search components, such as timeout, thread pool size and search handler pattern. ]
          Sabyasachi Dalal made changes -
          Attachment fedsearch.patch [ 12371498 ]
          Yonik Seeley made changes -
          Assignee Yonik Seeley [ yseeley@gmail.com ]
          Yonik Seeley made changes -
          Summary Federated Search over HTTP Distributed Search over HTTP
          Priority Minor [ 4 ] Major [ 3 ]
          Description Motivated by http://wiki.apache.org/solr/FederatedSearch
          "Index view consistency between multiple requests" requirement is relaxed in this implementation.

          Does the federated search query side. Update not yet done.

          Tries to achieve:-
          ------------------------
          - The client applications are totally agnostic to federated search. The federated search and merging of results are totally behind the scene in Solr in request handler . Response format remains the same after merging of results.
          The response from individual shard is deserialized into SolrQueryResponse object. The collection of SolrQueryResponse objects are merged to produce a single SolrQueryResponse object. This enables to use the Response writers as it is; or with minimal change.

          - Efficient query processing with highlighting and fields getting generated only for merged documents. The query is executed in 2 phases. First phase gets the doc unique keys with sort criteria. Second phase brings all requested fields and highlighting information. This saves lot of CPU in case there are good number of shards and highlighting info is requested.
          Should be easy to customize the query execution. For example: user can specify to execute query in just 1 phase itself. (For some queries when highlighting info is not required and number of fields requested are small; this can be more efficient.)

          - Ability to easily overwrite the default Federated capability by appropriate plugins and request parameters. As federated search is performed by the RequestHandler itself, multiple request handlers can easily be pre-configured with different federated search settings in solrconfig.xml

          - Global weight calculation is done by querying the terms' doc frequencies from all shards.

          - Federated search works on Http transport. So individual shard's VIP can be queried. Load-balancing and Fail-over taken care by VIP as usual.

          -Sub-searcher response parsing as a plugin interface. Different implementation could be written based on JSON, xml SAX etc. Current one based on XML DOM.


          HOW:
          -------
          A new RequestHandler called MultiSearchRequestHandler does the federated search on multiple sub-searchers, (referred as "shards" going forward). It extends the RequestHandlerBase. handleRequestBody method in RequestHandlerBase has been divided into query building and execute methods. This has been done to calculate global numDocs and docFreqs; and execute the query efficiently on multiple shards.
          All the "search" request handlers are expected to extend MultiSearchRequestHandler class in order to enable federated capability for the handler. StandardRequestHandler and DisMaxRequestHandler have been changed to extend this class.
           
          The federated search kicks in if "shards" is present in the request parameter. Otherwise search is performed as usual on the local index. eg. shards=local,host1:port1,host2:port2 will search on the local index and 2 remote indexes. The search response from all 3 shards are merged and serviced back to the client.

          The search request processing on the set of shards is performed as follows:

          STEP 1: The query is built, terms are extracted. Global numDocs and docFreqs are calculated by requesting all the shards and adding up numDocs and docFreqs from each shard.

          STEP 2: (FirstQueryPhase) All shards are queried. Global numDocs and docFreqs are passed as request parameters. All document fields are NOT requested, only document uniqFields and sort fields are requested. MoreLikeThis and Highlighting information are NOT requested.

          STEP 3: Responses from FirstQueryPhase are merged based on "sort", "start" and "rows" params. Merged doc uniqField and sort fields are collected. Other information like facet and debug is also merged.

          STEP 4: (SecondQueryPhase) Merged doc uniqFields and sort fields are grouped based on shards. All shards in the grouping are queried for the merged doc uniqFields (from FirstQueryPhase), highlighting and moreLikeThis info.

          STEP 5: Responses from all shards from SecondQueryPhase are merged.

          STEP 6: Document fields , highlighting and moreLikeThis info from SecondQueryPhase are merged into FirstQueryPhase response.




          TODO:
          -Support sort field other than default score
          -Support ResponseDocs in writers other than XMLWriter
          -Http connection timeouts

          OPEN ISSUES;
          -Merging of facets by "top n terms of field f"

          Scope for Performance optimization:-
          -Search shards in parallel threads
          -Http connection Keep-Alive ?
          -Cache global numDocs and docFreqs
          -Cache Query objects in handlers ??

          Would appreciate feedback on my approach. I understand that there would be lot things I might have over-looked.
          Searching over multiple shards and aggregating results.
          Motivated by http://wiki.apache.org/solr/DistributedSearch
          Sabyasachi Dalal made changes -
          Attachment fedsearch.patch [ 12369782 ]
          Sharad Agarwal made changes -
          Attachment fedsearch.patch [ 12366830 ]
          Stu Hood made changes -
          Attachment fedsearch.stu.patch [ 12366389 ]
          Stu Hood made changes -
          Attachment fedsearch.stu.patch [ 12366220 ]
          Sharad Agarwal made changes -
          Attachment fedsearch.patch [ 12365612 ]
          Sharad Agarwal made changes -
          Attachment fedsearch.patch [ 12362862 ]
          Sharad Agarwal made changes -
          Link This issue is blocked by SOLR-281 [ SOLR-281 ]
          Sharad Agarwal made changes -
          Field Original Value New Value
          Attachment fedsearch.patch [ 12361878 ]
          Sharad Agarwal created issue -

            People

            • Assignee:
              Yonik Seeley
              Reporter:
              Sharad Agarwal
            • Votes:
              14 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development