[ARROW-10251] [Rust] [DataFusion] MemTable::load() should load partitions in parallel

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: beginner pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26248
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 09/Oct/20 18:35
• Updated: 11/Jan/23 08:12
• Resolved: 11/Oct/20 16:41

Description

MemTable::load() should load partitions in parallel using async tasks, rather than loading one partition at a time.

Also, we should make batch size configurable. It is currently hard-coded to 1024*1024 which can be quite inefficient.

Issue Links

• links to: GitHub Pull Request #8428