Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-301

Add `Program` interface, a planner phase more general than current `RuleSet`

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8
    • Component/s: None
    • Labels:

      Description

      Currently `Framework.getPlanner` allows you to pass in an array of `RuleSet` objects. You can then invoke these one by one to get multi-phase optimization.

      A new `Program` interface generalizes `RuleSet`. It could, for instance, choose a different set of rules based on the input. It could invoke a particular planner engine (hep or volcano), and enable a particular set of traits. It could also invoke `RelNode` transformations that are ad hoc, not implemented in `RelOptRule`s.

      One of the current `Framework.getPlanner` methods is now deprecated and will be removed before optiq-0.9; new code should use a version that passes a list of `Program`.

      ---------------- Imported from GitHub ----------------
      Url: https://github.com/julianhyde/optiq/issues/301
      Created by: julianhyde
      Labels:
      Created at: Wed Jun 18 02:02:38 CEST 2014
      State: open

        Activity

        Hide
        github-import GitHub Import added a comment -

        [Date: Wed Jun 18 03:41:50 CEST 2014, Author: jacques-n]

        We leverage the Framework interface extensively and have a large amount of logic outside of it. I look forward to your specific proposal but want to make sure that we don't restrict flexibility around using the planner. Having to rewrite all our code to use a functional paradigm seems less than ideal and not as flexible.

        We actually spent a quite some time trying to come up with similar but slightly higher level procedural abstraction inside of Drill and found that the termination points and data sharing semantics were too varied. As such, we actually defined a base class that we then allowed subclasses of to solve the problem. For reference, that model can be seen here:

        the base class:
        https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java

        example subclasses:
        https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ExplainHandler.java
        https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateTableHandler.java

        You can see how individual subclasses choose to override or completely rework the flow when they want to deviate. They can also copy and paste portions of the base class' getPlan() as needed.

        Show
        github-import GitHub Import added a comment - [Date: Wed Jun 18 03:41:50 CEST 2014, Author: jacques-n ] We leverage the Framework interface extensively and have a large amount of logic outside of it. I look forward to your specific proposal but want to make sure that we don't restrict flexibility around using the planner. Having to rewrite all our code to use a functional paradigm seems less than ideal and not as flexible. We actually spent a quite some time trying to come up with similar but slightly higher level procedural abstraction inside of Drill and found that the termination points and data sharing semantics were too varied. As such, we actually defined a base class that we then allowed subclasses of to solve the problem. For reference, that model can be seen here: the base class: https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java example subclasses: https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/ExplainHandler.java https://github.com/apache/incubator-drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/CreateTableHandler.java You can see how individual subclasses choose to override or completely rework the flow when they want to deviate. They can also copy and paste portions of the base class' getPlan() as needed.
        Hide
        github-import GitHub Import added a comment -

        [Date: Wed Jun 18 03:53:30 CEST 2014, Author: julianhyde]

        Program literally generalizes RuleSet (in the sense that there is a RuleSetProgram). So your code should continue to work.

        I know that there will always be some patterns of use of a planner that are too complex to be done using what you call a functional paradigm. In those cases you can continue to invoke the planner from the outside.

        I was able to represent a moderately complex case: if join count < 6, use the regular rule set; otherwise invoke HepPlanner with a few rules to create MultiJoinRel, then invoke VolcanoPlanner with SwapJoinRule and PushJoinThroughJoin disabled and LoptOptimizeJoinRule enabled. I hope we’ll see a few more moderately complex cases, and can start to build re-usable programs that can be composed.

        My changes are currently in a branch: https://github.com/julianhyde/optiq/commit/2f52fb17d0a6e25c026baac37b4b96d0f45441a5. Review comments welcome.

        Show
        github-import GitHub Import added a comment - [Date: Wed Jun 18 03:53:30 CEST 2014, Author: julianhyde ] Program literally generalizes RuleSet (in the sense that there is a RuleSetProgram). So your code should continue to work. I know that there will always be some patterns of use of a planner that are too complex to be done using what you call a functional paradigm. In those cases you can continue to invoke the planner from the outside. I was able to represent a moderately complex case: if join count < 6, use the regular rule set; otherwise invoke HepPlanner with a few rules to create MultiJoinRel, then invoke VolcanoPlanner with SwapJoinRule and PushJoinThroughJoin disabled and LoptOptimizeJoinRule enabled. I hope we’ll see a few more moderately complex cases, and can start to build re-usable programs that can be composed. My changes are currently in a branch: https://github.com/julianhyde/optiq/commit/2f52fb17d0a6e25c026baac37b4b96d0f45441a5 . Review comments welcome.
        Show
        julianhyde Julian Hyde added a comment - Fixed in master branch https://github.com/julianhyde/optiq/commit/3e4fb981d107af9bdd9b1b4090e5de45b8b0f1c5 .
        Hide
        julianhyde Julian Hyde added a comment -

        Close issues resolved in 0.8 release (2014-06-27).

        Show
        julianhyde Julian Hyde added a comment - Close issues resolved in 0.8 release (2014-06-27).

          People

          • Assignee:
            Unassigned
            Reporter:
            github-import GitHub Import
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development