[ARROW-9726] [Rust] [DataFusion] ParquetScanExec launches threads too early

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Rust, Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25781
• Assignee: Andy Grove
• Reporter: Andy Grove
• Created: 13/Aug/20 14:59
• Updated: 11/Jan/23 08:08
• Resolved: 14/Aug/20 11:58

Description

ParquetScanExec launches threads in partitions() ahead of execute() being called on those partitions. This results on "too many open files" when there are many partitions.

Issue Links

• fixes: ARROW-6154 [Rust] [Parquet] Too many open files (os error 24)
• links to: GitHub Pull Request #7961