[HADOOP-14531] [Umbrella] Improve S3A error handling & reporting

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 3.1.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 16/Jun/17 11:03
• Updated: 22/Mar/18 05:17
• Resolved: 13/Feb/18 15:17

Description

Improve S3a error handling and reporting

this includes

1. looking at error codes and translating to more specific exceptions
2. better retry logic where present
3. adding retry logic where not present
4. more diagnostics in exceptions
5. docs

Overall goals

• things that can be retried and will go away are retried for a bit
• things that don't go away when retried failfast (302, no auth, unknown host, connection refused)
• meaningful exceptions are built in translate exception
• diagnostics are included, where possible
• our troubleshooting docs are expanded with new failures we encounter

AWS S3 error codes: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Issue Links

• depends upon: HADOOP-13811 s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
• depends upon: HADOOP-14012 Handled dynamo exceptions in translateException
• is depended upon by: HADOOP-13271 Intermittent failure of TestS3AContractRootDir.testListEmptyRootDirectory
• is depended upon by: HADOOP-14831 Über-jira: S3a phase IV: Hadoop 3.1 features
• is depended upon by: HADOOP-14825 Über-JIRA: S3Guard Phase II: Hadoop 3.1 features
• is duplicated by: HADOOP-14590 Extend and enhance S3A exception handling
• is related to: HADOOP-15076 Enhance S3A troubleshooting documents and add a performance document

Sub-Tasks

• Review retry logic on all S3 SDK calls, implement where needed: Resolved
• S3AUtils.translateException to map 503 reponse to => throttling failure: Resolved
• S3A to support custom retry policies; failfast on unknown host: Resolved
• S3AInputStream to use a retry policy on read failures: Resolved
• s3a troubleshooting to cover the "JSON parse error" message: Resolved
• handle InternalError in bulk object delete through retries: Resolved
• s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class: Resolved
• S3A GET/HEAD requests failing: java.lang.IllegalStateException: Connection is not open/Connection pool shut down: Resolved
• S3a over-reacts to potentially transient network problems in its init() logic: Resolved