Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5359

ClassCastException when push down filter on the output of flatten into parquet scan

    XMLWordPrintableJSON

    Details

      Description

      The following simplified query would hit ClassCastException.

      select n_regionkey
      from (select n_regionkey, 
                  flatten(nation.cities) as cities 
            from cp.`tpch/nation.parquet` nation) as flattenedCities 
      where flattenedCities.cities.`zip` = '12345';
      

      Here is the stacktrace for the Exception :

      caused by: java.lang.ClassCastException: org.apache.drill.common.expression.FunctionCall cannot be cast to org.apache.drill.common.expression.SchemaPath
      	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:170) ~[classes/:na]
      	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
      	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
      	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.doFunction(DrillOptiq.java:205) ~[classes/:na]
      	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:105) ~[classes/:na]
      	at org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) ~[classes/:na]
      	at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) ~[calcite-core-1.4.0-drill-r19.jar:1.4.0-drill-r19]
      	at org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:77) ~[classes/:na]
      	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter.doOnMatch(ParquetPushDownFilter.java:141) ~[classes/:na]
      	at org.apache.drill.exec.store.parquet.ParquetPushDownFilter$1.onMatch(ParquetPushDownFilter.java:68) ~[classes/:na]
      

      The cause of this problem: Parquet filter pushdown rule tries to push a filter expression containing item/flatten operators into parquet scan. However, the method DrillOptiq.toDrill() does not allow such expression (since "flatten" is not a scalar function).

      The solution is to disable pushing such filter expression. Even the rule allows, the underneath parquet metadata would not have the corresponding statistics; there is no point to considering push such filter expression.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jni Jinfeng Ni
                Reporter:
                jni Jinfeng Ni
                Reviewer:
                Aman Sinha
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: