[ARROW-10788] [C++] Make S3 recursive walks parallel

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26728
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 02/Dec/20 12:41
• Updated: 11/Jan/23 08:15
• Resolved: 08/Dec/20 12:18

Description

Doing a recursive S3 directory walk using GetFileInfo(Selector) currently lists all encountered directories serially, waiting for the results of one directory listing (or portion thereof) before launching the next one. Instead, we should use the Async APIs provided by the AWS SDK to parallelize HTTP requests as much as possible.

Issue Links

• is related to: ARROW-8884 [C++] Listing files with S3FileSystem is slow
• links to: GitHub Pull Request #8818