Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6371

Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical stage

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.13.0
    • Fix Version/s: Future
    • Labels:
      None

      Description

      FilterSetOpTransposeRule, ProjectSetOpTransposeRule are leveraged in DRILL-3855.
      They are used in HepPlanner, but if they additionally will be enabled in main logical planning stage for Volcano planner, more cases will be covered with these rules.
      For example:

      WITH year_total_1
           AS (SELECT c.r_regionkey    customer_id,
                      1 year_total
               FROM   cp.`tpch/region.parquet` c
               UNION ALL
               SELECT c.n_nationkey    customer_id,
                      1 year_total
               FROM   cp.`tpch/nation.parquet` c),
           year_total_2
           AS (SELECT c.r_regionkey    customer_id,
                      1 year_total
               FROM   cp.`tpch/region.parquet` c
               UNION ALL
               SELECT c.n_nationkey    customer_id,
                      1 year_total
               FROM   cp.`tpch/nation.parquet` c)
      SELECT count(t_w_firstyear.customer_id) as ct
      FROM   year_total_1 t_w_firstyear,
             year_total_2 t_w_secyear
      WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
       AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
      

      Current plan after performing rules:

      LogicalAggregate(group=[{}], ct=[COUNT($0)])
        LogicalProject(customer_id=[$0])
          LogicalFilter(condition=[AND(=($1, $3), >($1, 0), >($3, 0))])
            LogicalJoin(condition=[true], joinType=[inner])
              LogicalUnion(all=[true])
                LogicalProject(customer_id=[$1], year_total=[1])
                  EnumerableTableScan(table=[[cp, tpch/region.parquet]])
                LogicalProject(customer_id=[$1], year_total=[1])
                  EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
              LogicalUnion(all=[true])
                LogicalProject(customer_id=[$1], year_total=[1])
                  EnumerableTableScan(table=[[cp, tpch/region.parquet]])
                LogicalProject(customer_id=[$1], year_total=[1])
                  EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
      

      Since LogicalFilter isn't under LogicalUnion the FilterSetOpTransposeRule is not performed.
      FilterJoinRule from main Drill logical stage pushes LogicalFilter below, but the stage with FilterSetOpTransposeRule is already finished.
      That's why FilterSetOpTransposeRule and ProjectSetOpTransposeRule should be used in Drill main logical stage with Volcano planner.

      Currently using them in Volcano Planner can cause infinite loops - CALCITE-1271 (can be resolved after solving CALCITE-2223)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vitalii Vitalii Diravka
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: