[ARROW-11907] [C++] Use our own executor in S3FileSystem

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27748
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 08/Mar/21 19:22
• Updated: 11/Jan/23 08:22
• Resolved: 17/Mar/21 10:26

Description

We use the AWS SDK async APIs in some places in S3FileSystem, but all they do is spawn a separate thread and run the sync API in it.

Instead we should use our IO executor, which would:
1) put an upper bound on the number of threads started
2) (presumably) reduce latency by reusing threads instead of spawning a throwaway thread for each async call
3) allow for cancellation

Issue Links

• links to: GitHub Pull Request #9678