[ARROW-5227] [Rust] [DataFusion] Re-implement query execution with an extensible physical query plan

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21700
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 28/Apr/19 16:08
• Updated: 11/Jan/23 07:39
• Resolved: 09/Oct/19 08:11

Description

 This story (maybe it should have been an epic with hindsight) is to re-implement query execution in DataFusion using a physical plan that supports partitions and parallel execution.

This will replace the current query execution which happens directly from the logical plan.

The new physical plan is based on traits and is therefore extensible by other projects that use Arrow. For example, another project could add physical plans for distributed compute.

See design doc at https://docs.google.com/document/d/1ATZGIs8ry_kJeoTgmJjLrg6Ssb5VE7lNzWuz_4p6EWk/edit?usp=sharing for more info

Issue Links

• supercedes: ARROW-4759 [Rust] [DataFusion] It should be possible to share an execution context between threads
• links to: GitHub Pull Request #4220
• links to: GitHub Pull Request #4221

Sub-Tasks

• [Rust] [DataFusion] Create traits for phsyical query plan: Resolved
• [Rust] [DataFusion] Implement parallel execution for parquet scan: Resolved
• [Rust] [DataFusion] Implement parallel execution for CSV scan: Resolved
• [Rust] [DataFusion] Implement parallel execution for projection: Resolved
• [Rust] [DataFusion] Implement parallel execution for selection: Resolved
• [Rust] [DataFusion] Implement parallel execution for hash aggregate: Resolved
• [Rust] [DataFusion] Implement parallel execution for limit: Resolved
• [Rust] [DataFusion] Create physical plan from logical plan: Resolved
• [Rust] [DataFusion] Refactor TableProvider to return thread-safe BatchIterator: Resolved
• [Rust] [DataFusion] Create "merge" execution plan: Resolved
• [Rust] [DataFusion] Implement SUM aggregate expression: Resolved
• [Rust] [DataFusion] Implement MIN and MAX aggregate expressions: Resolved
• [Rust] [DataFusion] Implement COUNT aggregate expression: Resolved
• [Rust] [DataFusion] Implement AVG aggregate expression: Resolved
• [Rust] [DataFusion] Implement numeric literal expressions: Resolved
• [Rust] [DataFusion] Implement CAST expression: Resolved
• [Rust] [DataFusion] Implement physical expression for binary expressions: Resolved
• [Rust] [DataFusion] Update examples to use physical query plan: Resolved
• [Rust] [DataFusion] Update unit tests to use physical query plan: Closed
• [Rust] [DataFusion] Update integration tests to use physical plan: Resolved
• [Rust] [DataFusion] Remove execution of logical plan: Resolved