[DRILL-5359] ClassCastException when push down filter on the output of flatten into parquet scan - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.9.0
Fix Version/s: 1.11.0
Component/s: Query Planning & Optimization
Labels:
- ready-to-commit

Description

The following simplified query would hit ClassCastException.

select n_regionkey
from (select n_regionkey, 
            flatten(nation.cities) as cities 
      from cp.`tpch/nation.parquet` nation) as flattenedCities 
where flattenedCities.cities.`zip` = '12345';

Here is the stacktrace for the Exception :

caused by: java.lang.ClassCastException: org.apache.drill.common.expression.FunctionCall cannot be cast to org.apache.drill.common.expression.SchemaPath
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:170) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.doFunction(DrillOptiq.java:205) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:105) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
	at org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:77) ~[classes/:na]
	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter.doOnMatch(ParquetPushDownFilter.java:141) ~[classes/:na]
	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter$1.onMatch(ParquetPushDownFilter.java:68) ~[classes/:na]

The cause of this problem: Parquet filter pushdown rule tries to push a filter expression containing item/flatten operators into parquet scan. However, the method DrillOptiq.toDrill() does not allow such expression (since "flatten" is not a scalar function).

The solution is to disable pushing such filter expression. Even the rule allows, the underneath parquet metadata would not have the corresponding statistics; there is no point to considering push such filter expression.