[ARROW-4681] [Rust] [DataFusion] Implement parallel query execution using threads

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21210
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 26/Feb/19 05:41
• Updated: 11/Jan/23 07:35
• Resolved: 10/Mar/19 17:01

Description

I am planning on tackling this soon. The basic plan is to change the table scan() method to return multiple scanners (one per partition) so that the query execution can process each partition on a separate thread.

This will involve changing some of the DataFusion APIs to use Arc instead of Rc.

Issue Links

• links to: GitHub Pull Request #3849

Sub-Tasks

• [Rust] [DataFusion] It should be possible to share an execution context between threads: Closed
• [Rust] [DataFusion] Table.scan() should return one iterator per partition: Closed