[HADOOP-18990] S3A: retry on credential expiry

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 25/Nov/23 13:11
• Updated: 20/May/24 10:29

Description

Reported in AWS SDK https://github.com/aws/aws-sdk-java-v2/issues/3408

In RetryableStage execute method, the "AwsCredentails" does not attempt to renew if it has expired. Therefore, if a method called with the existing credential is expiring soon, the number of retry is less than intended due to the expiration of the credential.

The stack from this report doesn't show any error detail we can use to identify the 400 exception as something we should be retrying on. This could be due to the logging, or it could actually hold. we've have to generate some socket credentials, let them expire and then see how hadoop fs commands failed. Something to do by hand as an STS test to do this is probably slow. unless we expire all session credentials of a given role?. Could be good, would be traumatic for other test runs though.

software.amazon.awssdk.services.s3.model.S3Exception: The provided token has expired. (Service: S3, Status Code: 400, Request ID: 3YWKVBNJPNTXPJX2, Extended Request ID: GkR56xA0r/Ek7zqQdB2ZdP3wqMMhf49HH7hc5N2TAIu47J3HEk6yvSgVNbX7ADuHDy/Irhr2rPQ=)

Issue Links

• is related to: HADOOP-19181 S3A) IAMCredentialsProvider throttling results in AWS auth failures
• links to: https://github.com/aws-greengrass/aws-greengrass-nucleus/pull/1583/files
• links to: https://github.com/aws/aws-sdk-java-v2/issues/3408