[ARROW-10675] [C++] Support AWS S3 Web identity credentials

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.1, 2.0.0
• Fix Version/s: 5.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26628
• Assignee: Sahil Gupta
• Reporter: Paul Balanca
• Created: 21/Nov/20 13:10
• Updated: 11/Jan/23 08:14
• Resolved: 01/Jun/21 16:39

Description

It seems to me that Arrow only supports at the moment the "AssumeRole" AWS STS API, but not the other options offered:

• https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison
• https://sdk.amazonaws.com/cpp/api/LATEST/class_aws_1_1_auth_1_1_s_t_s_assume_role_web_identity_credentials_provider.html

I am clearly no security/infra expert, but it seems that the configuration "AssumeRoleWithWebIdentity" is used commonly in Kubernetes setups, and I believe it would be beneficial for Arrow C++ & Python library to support.

At the moment, a workaround is to call directly `aws sts` to generate a temporary session, but it is a fairly painful solution as the session expires, all PyArrow objects with an S3 filesystem (datasets, ...) need to be re-built with new credentials. 

Issue Links

• is required by: ARROW-16457 [Python] Support AWS S3 Web identity credentials
• links to: GitHub Pull Request #8977
• links to: GitHub Pull Request #10088