Druid 0.11 has newly built in capabilities called Expressions that can be used to push expression like projects/aggregates/filters.
In order to leverage this new feature, some changes need to be done to the Druid Calcite adapter.
This is a link to the current supported functions and expressions in Druid
As you can see from the Docs an expression can be an actual tree of operators,
Expression can be used with Filters, Projects, Aggregates, PostAggregates and
Having filters. For Filters will have new Filter kind called Filter expression.
FYI, you might ask can we push everything as Expression Filter the short answer
is no because, other kinds of Druid filters perform better when used, Hence
Expression filter is a plan B sort of thing. In order to push expression as
Projects and Aggregates we will be using Expression based Virtual Columns.
The major change is the merging of the logic of pushdown verification code and
the Translation of RexCall/RexNode to Druid Json, native physical language. The
main drive behind this redesign is the fact that in order to check if we can
push down a tree of expressions to Druid we have to compute the Druid Expression
String anyway. Thus instead of having 2 different code paths, one for pushdown
validation and one for Json generation we can have one function that does both.
For instance instead of having one code path to test and check if a given filter
can be pushed or not and then having a translation layer code, will have
one function that either returns a valid Druid Filter or null if it is not
possible to pushdown. The same idea will be applied to how we push Projects and
Aggregates, Post Aggregates and Sort.
Here are the main elements/Classes of the new design. First will be merging the logic of
Translation of Literals/InputRex/RexCall to a Druid physical representation.
Translate leaf RexNode to Valid pair Druid Column + Extraction functions if possible
In the other hand, in order to Convert Literals to Druid Literals will introduce
Main new functions used to pushdown nodes and Druid Json generation.
Filter pushdown verification and generates is done via
For project pushdown added
For Grouping pushdown added
For Aggregation pushdown added
For sort pushdown added
to generate expression
For Expression computation most of the work is done here
public static String toDruidExpression(
final RexNode rexNode,
final RelDataType inputRowType,
final DruidQuery druidRel
Thus user can implement custom expression converter based on the SqlOperator syntax and signature.
The Druid Query Class will provide
org.apache.calcite.adapter.druid.DruidQuery#getOperatorConversionMap which is a
map of SqlOperator to DruidSqlOperatorConverter.
Any feedback is welcome.