[SOLR-16235] Allow Solr SQL to use the drill Streaming Expression for high cardinality aggregations

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Parallel SQL
• Labels: RobustSQL
• Assignee: Joel Bernstein
• Reporter: Joel Bernstein
• Created: 03/Jun/22 13:31
• Updated: 22/Jun/22 17:36

Description

The latest versions of Streaming Expressions support the drill function which is designed for high cardinality aggregations (https://solr.apache.org/guide/8_6/stream-source-reference.html#drill). Drill allows users to push down a Streaming Expression into the export handler itself and emit aggregated tuples over the wire. Because drill still takes advantage of the sort order of the export handler it supports unlimited cardinality. Do to the massive performance improvements in the export handler in Solr 9, drill is almost as fast as facets. Drill is not a replacement for facet mode though, which is still faster in low to medium cardinality situations.

This improvement should be rather easy to implement but there are some questions about the design. One thought I had was to add a new mode called drill to the existing map_reduce and facet modes. This would preserve the existing map_reduce execution plan. The other approach is simply to always use drill in map_reduce mode aggregations.