Apache Drill
  1. Apache Drill
  2. DRILL-1273

In case of multiple storage plugins (with optimization rules), all the queries with "where" clause will fail.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 0.4.0, 0.5.0
    • Fix Version/s: 0.5.0
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      In case of multiple storage plugins with respective optimization rules (like predicate push down), optiq optimizer will call onMatch() method on all the sub classes of "StoragePluginOptimizerRule".

      In this case, all the queries with "where" clause will fail with ClassCastException.

        Activity

        B Anil Kumar created issue -
        B Anil Kumar made changes -
        Field Original Value New Value
        Attachment 0001-DRILL-1273-patch.patch [ 12660559 ]
        Hide
        B Anil Kumar added a comment -

        I have attached the patch.

        Please review the patch.

        Show
        B Anil Kumar added a comment - I have attached the patch. Please review the patch.
        B Anil Kumar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jacques Nadeau made changes -
        Assignee DrillCommitter [ drillcommitter ]
        Hide
        Jacques Nadeau added a comment -

        looks fine. We should really consider refactoring this to a more general rule infrastructure in the future.

        Show
        Jacques Nadeau added a comment - looks fine. We should really consider refactoring this to a more general rule infrastructure in the future.
        Hide
        Jacques Nadeau added a comment -

        Actually, this shouldn't be needed. The matches() method already does this check. When were you experiencing this problem as that should filter out any other matches?

        Show
        Jacques Nadeau added a comment - Actually, this shouldn't be needed. The matches() method already does this check. When were you experiencing this problem as that should filter out any other matches?
        Jacques Nadeau made changes -
        Assignee DrillCommitter [ drillcommitter ] B Anil Kumar [ akumarb2010 ]
        Hide
        B Anil Kumar added a comment - - edited

        Thanks for pointing it.

        My bad. As part of Mongo predicate pushdown implementation, we implemented MongoPushDownFilterForScan but we missed to override matches() method.

        In this case, when we fire hbase query with where clause like "SELECT * FROM hbase.`students` where row_key = 'student1' " then It is also going into onMatch of MongoPushDownFilterForScan. This is due to missing matches() implementation in MongoPushDownFilterForScan.

        014-08-12 10:29:15,957 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] DEBUG o.a.d.e.store.hbase.HBaseGroupScan - Getting region locations
        2014-08-12 10:29:21,154 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 748e4c1a-19c0-4aa4-bb12-f86ee3220c98: Failure while setting up Foreman.
        java.lang.ClassCastException: org.apache.drill.exec.store.hbase.HBaseGroupScan cannot be cast to org.apache.drill.exec.store.mongo.MongoGroupScan
        	at org.apache.drill.exec.store.mongo.MongoPushDownFilterForScan.onMatch(MongoPushDownFilterForScan.java:52) ~[drill-mongo-storage-0.4.0-incubating-SNAPSHOT.jar:0.4.0-incubating-SNAPSHOT]
        	at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:273) ~[optiq-core-0.9-20140730.000241-5.jar:na]
        	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:157) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:403) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250) [drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT]
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
        	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
        

        I haven't observed matches() implementation of HBasePushFilterIntoScan and thought similar issue might present in HBase. That's why I raised this issue. So please invalidate this issue.

        But I have few doubts. Can you please explain the below?

        1) Storage optimization rules are specific to each storage plugin, So we should only call onMatch() of below instance right? Why we should even call matches() of each optimization rule of different storage plugins?

         public Set<StoragePluginOptimizerRule> getOptimizerRules() {
            return ImmutableSet.of(HBasePushFilterIntoScan.INSTANCE);
          }
        

        2) Why default matches() method returning true? Is it a good idea to override it with false as default value in StoragePluginOptimizerRule?

        Show
        B Anil Kumar added a comment - - edited Thanks for pointing it. My bad. As part of Mongo predicate pushdown implementation, we implemented MongoPushDownFilterForScan but we missed to override matches() method. In this case, when we fire hbase query with where clause like "SELECT * FROM hbase.`students` where row_key = 'student1' " then It is also going into onMatch of MongoPushDownFilterForScan. This is due to missing matches() implementation in MongoPushDownFilterForScan. 014-08-12 10:29:15,957 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] DEBUG o.a.d.e.store.hbase.HBaseGroupScan - Getting region locations 2014-08-12 10:29:21,154 [da4d10d7-d4ab-457d-b648-5c23ad6ffdf6:foreman] ERROR o.a.drill.exec.work.foreman.Foreman - Error 748e4c1a-19c0-4aa4-bb12-f86ee3220c98: Failure while setting up Foreman. java.lang.ClassCastException: org.apache.drill.exec.store.hbase.HBaseGroupScan cannot be cast to org.apache.drill.exec.store.mongo.MongoGroupScan at org.apache.drill.exec.store.mongo.MongoPushDownFilterForScan.onMatch(MongoPushDownFilterForScan.java:52) ~[drill-mongo-storage-0.4.0-incubating-SNAPSHOT.jar:0.4.0-incubating-SNAPSHOT] at org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223) ~[optiq-core-0.9-20140730.000241-5.jar:na] at org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661) ~[optiq-core-0.9-20140730.000241-5.jar:na] at net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165) ~[optiq-core-0.9-20140730.000241-5.jar:na] at net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:273) ~[optiq-core-0.9-20140730.000241-5.jar:na] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:157) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:128) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:403) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:219) ~[drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:250) [drill-java-exec-0.4.0-incubating-SNAPSHOT-rebuffed.jar:0.4.0-incubating-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] I haven't observed matches() implementation of HBasePushFilterIntoScan and thought similar issue might present in HBase. That's why I raised this issue. So please invalidate this issue. But I have few doubts. Can you please explain the below? 1) Storage optimization rules are specific to each storage plugin, So we should only call onMatch() of below instance right? Why we should even call matches() of each optimization rule of different storage plugins? public Set<StoragePluginOptimizerRule> getOptimizerRules() { return ImmutableSet.of(HBasePushFilterIntoScan.INSTANCE); } 2) Why default matches() method returning true? Is it a good idea to override it with false as default value in StoragePluginOptimizerRule?
        B Anil Kumar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Invalid [ 6 ]
        Jacques Nadeau made changes -
        Fix Version/s 0.5.0 [ 12324880 ]
        Tony Stevenson made changes -
        Workflow no-reopen-closed, patch-avail, testing [ 12881369 ] Drill workflow [ 12934504 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4m 35s 1 B Anil Kumar 08/Aug/14 07:00
        Patch Available Patch Available Resolved Resolved
        5d 4m 1 B Anil Kumar 13/Aug/14 07:04

          People

          • Assignee:
            B Anil Kumar
            Reporter:
            B Anil Kumar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development