[HADOOP-13205] S3A to support custom retry policies; failfast on unknown host

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.8.0
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/May/16 14:02
• Updated: 22/Mar/18 04:47
• Resolved: 22/Mar/18 04:47

Description

Noticed today that when connections are down, S3A retries on UnknownHostExceptions logging noisily in the process.

1. it should be possible to define or customize retry policies for an FS instance (fail fast, exponential backoff, etc)
2. we may want to explicitly have a fail-fast-if-offline retry policy, catching the common connectivity ones.

Testing will be fun here.

Issue Links

• is duplicated by: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• is part of: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints
• is related to: HADOOP-14303 Review retry logic on all S3 SDK calls, implement where needed
• is related to: HADOOP-14564 s3a test can hang in teardown with network problems