Apache Drill
  1. Apache Drill
  2. DRILL-491

Support project push-down into scan operator

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, the scan operator will read all the columns from the data source, and pass all the columns to down-stream operator.

      In order to improve performance of Drill query, we should instead push the project operation into scan operator. The scan operator will take as an input a list of columns required to retrieve. If the query only refers a subset of columns, the scan operator will only have to retrieve values for the column in the list.

      It's expected to see performance improvement once project push-down is supported.

        Issue Links

          Activity

          Jinfeng Ni created issue -
          Hide
          Jinfeng Ni added a comment -

          This patch is basing on tpch_work branch work.

          Show
          Jinfeng Ni added a comment - This patch is basing on tpch_work branch work.
          Jinfeng Ni made changes -
          Field Original Value New Value
          Attachment DRILL-491.1.patch [ 12639102 ]
          Hide
          Jacques Nadeau added a comment -

          please update patch for latest master

          Show
          Jacques Nadeau added a comment - please update patch for latest master
          Jinfeng Ni made changes -
          Attachment DRILL-491.1.patch [ 12639102 ]
          Hide
          Jinfeng Ni added a comment -

          The patch has the following change:

          1. Rebase on latest master branch.

          2. Move push project rule into Drill Logical planning phase.

          3. Add DrillPushProjIntoScan rule.

          Show
          Jinfeng Ni added a comment - The patch has the following change: 1. Rebase on latest master branch. 2. Move push project rule into Drill Logical planning phase. 3. Add DrillPushProjIntoScan rule.
          Jinfeng Ni made changes -
          Attachment DRILL-491.2.patch [ 12642157 ]
          Jinfeng Ni made changes -
          Attachment DRILL-491.2.patch [ 12642157 ]
          Hide
          Jinfeng Ni added a comment -

          The patch contains the following change:

          1. Revise code based on review comments.

          2. Add method clone() to GroupScan interface. This will save the time spent on initializing groupscan instance, when optiq rules are fired multiple times.

          3. Add a base class for plan testing. The base class will be used for verifying either logical / physical plan produced by optimizer.

          4. Add more test to test project pushdown : use INFORMATION_SCHEMA (for schema-based query), and TPCH similar queries.

          Show
          Jinfeng Ni added a comment - The patch contains the following change: 1. Revise code based on review comments. 2. Add method clone() to GroupScan interface. This will save the time spent on initializing groupscan instance, when optiq rules are fired multiple times. 3. Add a base class for plan testing. The base class will be used for verifying either logical / physical plan produced by optimizer. 4. Add more test to test project pushdown : use INFORMATION_SCHEMA (for schema-based query), and TPCH similar queries.
          Jinfeng Ni made changes -
          Attachment DRILL-491.3.patch [ 12642806 ]
          Hide
          Jinfeng Ni added a comment -

          DRILL-491 is currently basing on DRILL-571, which bases on DRILL-494.

          Show
          Jinfeng Ni added a comment - DRILL-491 is currently basing on DRILL-571 , which bases on DRILL-494 .
          Hide
          Jacques Nadeau added a comment -

          merged in a3d1472

          Show
          Jacques Nadeau added a comment - merged in a3d1472
          Jacques Nadeau made changes -
          Link This issue is broken by DRILL-631 [ DRILL-631 ]
          Jacques Nadeau made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jake Farrell made changes -
          Workflow no-reopen-closed, patch-avail [ 12854497 ] no-reopen-closed, patch-avail, testing [ 12860186 ]
          Jacques Nadeau made changes -
          Fix Version/s 0.4.0 [ 12324963 ]
          Tony Stevenson made changes -
          Workflow no-reopen-closed, patch-avail, testing [ 12860186 ] Drill workflow [ 12934231 ]

            People

            • Assignee:
              Jinfeng Ni
              Reporter:
              Jinfeng Ni
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development