[ARROW-16457] [Python] Support AWS S3 Web identity credentials

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/20228
• Assignee: Unassigned
• Reporter: Antoine Pitrou
• Created: 04/May/22 13:26
• Updated: 11/Jan/23 11:44

Description

ARROW-10675 added support for AWS S3 Web identity credentials on the C++ side. We should bind that functionality on the Python side.

To avoid proliferation of authentication arguments to the S3FileSystem constructor, some of which mutually exclusive (but not all), we should probably add instead a more flexible auth argument that could represent to different authentication kinds.

There is a bit of API design necessary. IMHO it's probably best if the auth argument is a dedicated S3Auth object with several constructors, but perhaps we can instead admit some kind of dict?

Issue Links

• requires: ARROW-10675 [C++] Support AWS S3 Web identity credentials