[ARROW-6691] [Rust] [DataFusion] Use tokio and Futures instead of spawning threads

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Fix Version/s: 1.0.0
• Component/s: Rust, Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/23038
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 25/Sep/19 13:59
• Updated: 11/Jan/23 07:48
• Resolved: 12/May/20 14:16

Description

The current implementation of the physical query plan uses "thread::spawn" which is expensive. We should switch to using Futures, async!/await!, and tokio so that we are launching tasks in a thread pool instead and writing idiomatic Rust code with futures combinators to chain actions together.