[ARROW-8774] [Rust] [DataFusion] Improve threading model

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• External issue URL: https://github.com/apache/arrow/issues/24921
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 12/May/20 14:21
• Updated: 11/Jan/23 08:02
• Resolved: 14/Jul/20 13:49

Description

DataFusion currently spawns one thread per partition and this results in poor performance if there are more partitions than available cores/threads. It would be better to have a thread-pool that defaults to number of available cores.

Here is a Google doc where we can collaborate on a design discussion.

https://docs.google.com/document/d/1_wc6diy3YrRgEIhVIGzrO5AK8yhwfjWlmKtGnvbsrrY/edit?usp=sharing