[ARROW-9732] [Rust] [DataFusion] Add "Physical Planner" type thing which can do optimizations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• External issue URL: https://github.com/apache/arrow/issues/25786
• Assignee: Unassigned
• Reporter: Andrew Lamb
• Created: 13/Aug/20 19:15
• Updated: 11/Jan/23 08:08
• Resolved: 17/Aug/20 12:00

Description

andygrove implemented what I would describe as a "limit pushdown" optimization within Limit here: https://github.com/apache/arrow/pull/7958#discussion_r470175966

However, it was implemented by directly instantiating Partition objects during plan execution. This "pick the top N from each partition and then pick the top N from the merged result" is an example of operator pushdown that could be done at planning time

This ticket tracks the work to add some way to represent the in the planning stage, rather than execution, in order to open up more optimization opportunities.

One example of pushdown that could potentially happen at planning time would be pushing the limit down past Projections for example.

Issue Links

• duplicates: ARROW-9758 [Rust] [DataFusion] Implement extension API for DataFusion
• is related to: ARROW-9464 [Rust] [DataFusion] Physical plan refactor to support optimization rules and more efficient use of threads