Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.3.0
Description
Although the planner does much to reduce the output cardinality of operators where possible (e.g. by predicate pushdown), there are some situations where the planner cannot apply a selectivity-based optimisation, but the execution engine can based on the output from operators at runtime. One example is 'dynamic partition pruning', where a filter is computed on a join key which is a partition column by the build side of a hash table, and then propagated to the probe side scan. The scan can then simply skip those partitions which don't appear in the filters.
There are other opportunities for optimisation based on runtime-computed filters. This JIRA tracks adding a general mechanism for filter computation and propagation in the backend, and the corresponding work in the planner to identify valid optimisation opportunities.
Attachments
Issue Links
- causes
-
IMPALA-10252 Query returns less number of rows with run-time filtering on integer column in a subquery against functional_parquet schema
- Resolved