[ARROW-8137] [C++][Dataset] Investigate multithreaded discovery

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/24343
• Assignee: Unassigned
• Reporter: Ben Kietzman
• Created: 17/Mar/20 14:23
• Updated: 11/Jan/23 07:58

Description

Currently FileSystemDatasetFactory Inpsects all files serially. For slow file systems or systems which support batched reads, this could be accelerated by inspecting files in parallel.

Issue Links

• is related to: ARROW-15724 [C++] Reduce directory and file IO when reading partition parquet dataset with partition key filters