[HADOOP-16499] S3A retry policy to be exponential

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.1.2
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Release Note: The S3A filesystem now backs off exponentially on failures. if you have customized the fs.s3a.retry.limit and fs.s3a.retry.interval options you may wish to review these settings
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 07/Aug/19 15:42
• Updated: 20/Aug/19 16:57
• Resolved: 09/Aug/19 15:32

Description

the fixed s3a retry policy doesnt leave big enough gaps for cached 404s to expire; we cant recover from this

HADOOP-16490 is a full fix for this, but one we can backport is moving from fixed to exponential retries

Issue Links

• is related to: HADOOP-16490 Avoid/handle cached 404s during S3A file creation
• links to: GitHub Pull Request #1246