Apache Drill
  1. Apache Drill
  2. DRILL-1273

In case of multiple storage plugins (with optimization rules), all the queries with "where" clause will fail.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.4.0, 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      In case of multiple storage plugins with respective optimization rules (like predicate push down), optiq optimizer will call onMatch() method on all the sub classes of "StoragePluginOptimizerRule".

      In this case, all the queries with "where" clause will fail with ClassCastException.

        Activity

        Hide
        B Anil Kumar added a comment -

        I have attached the patch.

        Please review the patch.

        Show
        B Anil Kumar added a comment - I have attached the patch. Please review the patch.
        Hide
        Jacques Nadeau added a comment -

        looks fine. We should really consider refactoring this to a more general rule infrastructure in the future.

        Show
        Jacques Nadeau added a comment - looks fine. We should really consider refactoring this to a more general rule infrastructure in the future.
        Hide
        Jacques Nadeau added a comment -

        Actually, this shouldn't be needed. The matches() method already does this check. When were you experiencing this problem as that should filter out any other matches?

        Show
        Jacques Nadeau added a comment - Actually, this shouldn't be needed. The matches() method already does this check. When were you experiencing this problem as that should filter out any other matches?
        Hide
        B Anil Kumar added a comment - - edited

        Thanks for pointing it.

        My bad. As part of Mongo predicate pushdown implementation, we implemented MongoPushDownFilterForScan but we missed to override matches() method.

        In this case, when we fire hbase query with where clause like "SELECT * FROM hbase.`students` where row_key = 'student1' " then It is also going into onMatch of MongoPushDownFilterForScan. This is due to missing matches() implementation in MongoPushDownFilterForScan.

        014-08-12 10:29:15,957 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] DEBUG o.a.d.e.store.hbase.HBaseGroupScan - Getting region locations
        2014-08-12 10:29:21,154 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 748e4c1a-19c0-4aa4-bb12-f86ee3220c98: Failure while setting up Foreman.
        java.lang.ClassCastException: org.apache.drill.exec.store.hbase.HBaseGroupScan cannot be cast to org.apache.drill.exec.store.mongo.MongoGroupScan
        	at org.apache.drill.exec.store.mongo.MongoPushDownFilterForScan.onMatch(MongoPushDownFilterForScan.java:52) ~[drill-mongo-storage-0.4.0-incubating-SNAPSHOT.jar:0.4.0-incubating-SNAPSHOT]
        	at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:273) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:157) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:403) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250) [drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
        	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
        

        I haven't observed matches() implementation of HBasePushFilterIntoScan and thought similar issue might present in HBase. That's why I raised this issue. So please invalidate this issue.

        But I have few doubts. Can you please explain the below?

        1) Storage optimization rules are specific to each storage plugin, So we should only call onMatch() of below instance right? Why we should even call matches() of each optimization rule of different storage plugins?

         public Set<StoragePluginOptimizerRule> getOptimizerRules() {
            return ImmutableSet.of(HBasePushFilterIntoScan.INSTANCE);
          }
        

        2) Why default matches() method returning true? Is it a good idea to override it with false as default value in StoragePluginOptimizerRule?

        Show
        B Anil Kumar added a comment - - edited Thanks for pointing it. My bad. As part of Mongo predicate pushdown implementation, we implemented MongoPushDownFilterForScan but we missed to override matches() method. In this case, when we fire hbase query with where clause like "SELECT * FROM hbase.`students` where row_key = 'student1' " then It is also going into onMatch of MongoPushDownFilterForScan. This is due to missing matches() implementation in MongoPushDownFilterForScan. 014-08-12 10:29:15,957 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] DEBUG o.a.d.e.store.hbase.HBaseGroupScan - Getting region locations 2014-08-12 10:29:21,154 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 748e4c1a-19c0-4aa4-bb12-f86ee3220c98: Failure while setting up Foreman. java.lang.ClassCastException: org.apache.drill.exec.store.hbase.HBaseGroupScan cannot be cast to org.apache.drill.exec.store.mongo.MongoGroupScan at org.apache.drill.exec.store.mongo.MongoPushDownFilterForScan.onMatch(MongoPushDownFilterForScan.java:52) ~[drill-mongo-storage-0.4.0-incubating-SNAPSHOT.jar:0.4.0-incubating-SNAPSHOT] at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223) ~[optiq-core-0.9-20140730.000241-5.jar:na] at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661) ~[optiq-core-0.9-20140730.000241-5.jar:na] at net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165) ~[optiq-core-0.9-20140730.000241-5.jar:na] at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:273) ~[optiq-core-0.9-20140730.000241-5.jar:na] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:157) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:403) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250) [drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] I haven't observed matches() implementation of HBasePushFilterIntoScan and thought similar issue might present in HBase. That's why I raised this issue. So please invalidate this issue. But I have few doubts. Can you please explain the below? 1) Storage optimization rules are specific to each storage plugin, So we should only call onMatch() of below instance right? Why we should even call matches() of each optimization rule of different storage plugins? public Set<StoragePluginOptimizerRule> getOptimizerRules() { return ImmutableSet.of(HBasePushFilterIntoScan.INSTANCE); } 2) Why default matches() method returning true? Is it a good idea to override it with false as default value in StoragePluginOptimizerRule?

          People

          • Assignee:
            B Anil Kumar
            Reporter:
            B Anil Kumar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development