Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-9732

[Rust] [DataFusion] Add "Physical Planner" type thing which can do optimizations

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Andy Grove implemented what I would describe as a "limit pushdown" optimization within Limit here: https://github.com/apache/arrow/pull/7958#discussion_r470175966

      However, it was implemented by directly instantiating Partition objects during plan execution. This "pick the top N from each partition and then pick the top N from the merged result" is an example of operator pushdown that could be done at planning time

      This ticket tracks the work to add some way to represent the in the planning stage, rather than execution, in order to open up more optimization opportunities.

      One example of pushdown that could potentially happen at planning time would be pushing the limit down past Projections for example.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                alamb Andrew Lamb
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: