[HADOOP-12353] S3 Native filesystem does not retry all connection failures

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.7.1
• Component/s: fs/s3
• Assignee: Mariusz Strzelecki
• Reporter: Mariusz Strzelecki
• Created: 25/Aug/15 11:59
• Updated: 13/Feb/16 15:00
• Resolved: 09/Feb/16 22:28

Description

Current implementation of NativeS3FileSystem.java uses RetryProxy that retries exceptions that may occur on network communication with S3 API, but these exceptions must be exact instances of IOException:
https://github.com/apache/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java#L349

Our tests show that HttpClient throws IOException subclasses which are not handled by Proxy.

Additionally, not all methods that call S3 API are listed to be handled, i.e. storeEmptyFile and retrieveMetadata are missing.