[ARROW-11924] [C++] Provide streaming output from GetFileInfo

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18555
• Assignee: Antoine Pitrou
• Reporter: Ben Kietzman
• Created: 10/Mar/21 16:06
• Updated: 11/Jan/23 08:23
• Resolved: 14/Apr/21 10:04

Description

For situations where a monolithic call to GetFileInfo will be slow, it would be useful to immediately receive any results which are ready through an AsyncGenerator<std::vector<FileInfo>> or so. This is probably a prerequisite of ARROW-8163, where the goal is to begin scanning known fragments while other fragments are still being discovered.

IIUC, one concrete example would be paging through a long output from S3's ListObjectsV2.

Issue Links

• is related to: ARROW-11935 [C++] Add push generator
• links to: GitHub Pull Request #9995