[ARROW-17306] [C++] Provide an optimized`GetFileInfoGenerator` specialization for `LocalFileSystem`

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32583

Language

C++

• Assignee: Pavel Solodovnikov
• Reporter: Pavel Solodovnikov
• Created: 04/Aug/22 09:25
• Updated: 11/Jan/23 11:49
• Resolved: 18/Aug/22 19:17

Description

At the moment, `LocalFileSystem` does not have a separate optimized implementation of `GetFileInfoGenerator` with a fallback to the generic `FileSystem::GetFileInfoGenerator`, which simply queues the synchronous version of `GetFileInfo(FileSelector)` to the background thread and waits for its completion before yielding.

This generally defeats all the purpose of `GetFileInfoGenerator` so that we cannot really use it to push down the `FileInfo` items to whatever consumer "on the fly" (e.g. `FileSystemDatasetFactory` and `FileSystemDataset`, correspondingly).

Provide a fair implementation so that it yields more than one time and allows to retrieve the data in chunks, so that the resulting `FileInfoGenerator` is usable for the purpose of streaming processing of data.

 

Issue Links

• links to: GitHub Pull Request #13796