[SOLR-6398] Add IterativeMergeStrategy to support running Parallel Iterative Algorithms inside of Solr

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 6.0
• Assignee: Unassigned
• Reporter: Joel Bernstein
• Created: 20/Aug/14 21:42
• Updated: 08/Oct/19 14:50
• Resolved: 16/Dec/15 19:48

Description

This ticket builds on the existing AnalyticsQuery / MergeStrategy framework by adding the abstract class IterativeMergeStrategy, which has built-in support for call-backs to the shards. The IterativeMergeStrategy is designed to support the execution of Parallel iterative Algorithms, such as Gradient Descent, inside of Solr.

To use the IterativeMergeStrategy you extend it and implement process(). This gives you access to the callback() method which makes it easy to make repeated calls to all the shards and run algorithms that require iteration.

Below is an example of a class that extends IterativeMergeStrategy. In this example it collects the count from the shards and then calls back to shards executing the !count AnalyticsQuery and sending it merged counts from all the shards.

class TestIterative extends IterativeMergeStrategy  {

    public void process(ResponseBuilder rb, ShardRequest sreq) throws Exception {
      int count = 0;
      for(ShardResponse shardResponse : sreq.responses) {
        NamedList response = shardResponse.getSolrResponse().getResponse();
        NamedList analytics = (NamedList)response.get("analytics");
        Integer c = (Integer)analytics.get("mycount");
        count += c.intValue();
      }

      ModifiableSolrParams params = new ModifiableSolrParams();
      params.add("distrib", "false");
      params.add("fq","{!count base="+count+"}");
      params.add("q","*:*");


      /*
      *  Call back to all the shards in the response and process the result.
       */

      QueryRequest request = new QueryRequest(params);
      List<Future<CallBack>> futures = callBack(sreq.responses, request);

      int nextCount = 0;

      for(Future<CallBack> future : futures) {
        QueryResponse response = future.get().getResponse();
        NamedList analytics = (NamedList)response.getResponse().get("analytics");
        Integer c = (Integer)analytics.get("mycount");
        nextCount += c.intValue();
      }

      NamedList merged = new NamedList();
      merged.add("mycount", nextCount);
      rb.rsp.add("analytics", merged);
    }
  }