[ARROW-8163] [C++][Dataset] Allow FileSystemDataset's file list to be lazy

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/24365
• Assignee: Unassigned
• Reporter: Ben Kietzman
• Created: 19/Mar/20 15:41
• Updated: 11/Jan/23 07:58

Description

A FileSystemDataset currently requires a full listing of files it contains on construction, so a scan cannot start until all files in the dataset are discovered. Instead it would be ideal if a large dataset could be constructed with a lazy file listing so that scans can start immediately.

Sub-Tasks

• [C++] Provide an optimized`GetFileInfoGenerator` specialization for `LocalFileSystem`: Resolved
• [C++][Dataset] Support async streaming interface for getting fragments in Dataset: Resolved
• [C++][Dataset] Allow creating FileSystemDataset with FileInfoGenerator as a source: Open