Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14481

Add drill Streaming Expression for efficient and accurate high cardinality aggregation

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: None
    • Fix Version/s: 8.6
    • Component/s: streaming expressions
    • Labels:
      None

      Description

      This ticket will add the drill Streaming Expression. The drill Streaming Expression is a wrapper around the functionality that is described in SOLR-14470. The idea is for drill to contact the /export handler on one replica in each shard of a collection and pass four parameters:

      • q: query
      • fl: field list
      • sort: sort spec
      • expr: The Streaming Expression sent to the /export handler to be executed.

      The export handler will pass the result set through the streaming expression performing an aggregation on the sorted result set and return the aggregated tuples. The drill expression will simply maintain the sort order of the tuples and emit them so that a wrapper expression can perform operations on the sorted aggregate tuples.

      Sample syntax:

      drill(collection1, q="*:*", fl="a,b,c", sort="a desc, b desc", rollup(input(), over="a,b", sum(c))) 

       In order to finish the aggregation other expressions can be used:

      rollup(
          select(
             drill(collection1, 
                   q="*:*", 
                   fl="a,b,c", 
                   sort="a desc, b desc", 
                   rollup(input(), over="a,b", sum(c))),
             a,
             b,
             sum(c) as sums),
          over="a, b",
          sum(sums))
             
       

       This provides fast aggregation over fields with infinite cardinality by pushing down the first level of aggregation into the /export handler.

       

       

        Attachments

        1. SOLR-14481.patch
          15 kB
          Joel Bernstein
        2. SOLR-14481.patch
          20 kB
          Joel Bernstein

          Issue Links

            Activity

              People

              • Assignee:
                jbernste Joel Bernstein
                Reporter:
                jbernste Joel Bernstein
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: