[ARROW-9716] [Rust] [DataFusion] MergeExec should have concurrency limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25771
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 13/Aug/20 00:19
• Updated: 11/Jan/23 08:08
• Resolved: 14/Aug/20 16:37

Description

MergeExec currently spins up one thread per input partition which causes apps to effectively hang if there are substantially more partitions than available cores.

We can implement a configurable limit here pretty easily.

Issue Links

• links to: GitHub Pull Request #7951