[ARROW-10366] [Rust] [DataFusion] Remove collect from merge

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26352
• Assignee: Andrew Lamb
• Reporter: Jorge Leitão
• Created: 22/Oct/20 05:09
• Updated: 11/Jan/23 08:12
• Resolved: 14/Nov/20 12:40

Description

The merge currently uses a collect within the thread, which causes the accumulation to happen before merging all batches together, see https://github.com/apache/arrow/pull/8473/files#r506304982

 

Issue Links

• links to: GitHub Pull Request #8503
• links to: GitHub Pull Request #8553