[HADOOP-18132] S3A to support exponential backoff when throttled

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Holden Karau
• Created: 18/Feb/22 21:08
• Updated: 21/Feb/22 14:49
• Resolved: 21/Feb/22 07:11

Description

S3 API has limits which we can exceed when using a large number of writers/readers/or listers. We should add randomized-exponential back-off to the s3 client when it encounters:

 

com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;