[HADOOP-19181] S3A) IAMCredentialsProvider throttling results in AWS auth failures

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 20/May/24 10:01
• Updated: 3 days ago 11:50

Description

Tests report throttling errors in IAM being remapped to noauth and failure

Again, impala tests, but with multiple processes on same host. this means that HADOOP-18945 isn't sufficient as even if it ensures a singleton instance for a process

• it doesn't if there are many test buckets (fixable)
• it doesn't work across processes (not fixable)

we may be able to

• use a singleton across all filesystem instances
• once we know how throttling is reported, handle it through retries + error/stats collection

Issue Links

• relates to: HADOOP-18990 S3A: retry on credential expiry
• relates to: HADOOP-18945 S3A: IAMInstanceCredentialsProvider failing: Failed to load credentials from IMDS
• relates to: IMPALA-13156 S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails to provide credentials for S3 access
• links to: https://github.com/aws/aws-sdk-java-v2/issues/3846
• links to: https://github.com/aws/aws-sdk-java-v2/issues/5247