[DRILL-5359] ClassCastException when push down filter on the output of flatten into parquet scan

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.11.0
• Component/s: Query Planning & Optimization
• Labels: ready-to-commit
• Assignee: Jinfeng Ni
• Reporter: Jinfeng Ni
• Reviewer: Aman Sinha
• Created: 16/Mar/17 21:40
• Updated: 20/Mar/17 19:54
• Resolved: 20/Mar/17 19:54

Description

The following simplified query would hit ClassCastException.

select n_regionkey
from (select n_regionkey, 
            flatten(nation.cities) as cities 
      from cp.`tpch/nation.parquet` nation) as flattenedCities 
where flattenedCities.cities.`zip` = '12345';

Here is the stacktrace for the Exception :

caused by: java.lang.ClassCastException: org.apache.drill.common.expression.FunctionCall cannot be cast to org.apache.drill.common.expression.SchemaPath
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:170) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.doFunction(DrillOptiq.java:205) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:105) ~[classes/:na]
	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
	at org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:77) ~[classes/:na]
	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter.doOnMatch(ParquetPushDownFilter.java:141) ~[classes/:na]
	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter$1.onMatch(ParquetPushDownFilter.java:68) ~[classes/:na]

The cause of this problem: Parquet filter pushdown rule tries to push a filter expression containing item/flatten operators into parquet scan. However, the method DrillOptiq.toDrill() does not allow such expression (since "flatten" is not a scalar function).

The solution is to disable pushing such filter expression. Even the rule allows, the underneath parquet metadata would not have the corresponding statistics; there is no point to considering push such filter expression.

Issue Links

• links to: GitHub Pull Request #786