[HADOOP-15216] S3AInputStream to handle reconnect on read() failure better

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 08/Feb/18 18:41
• Updated: 19/Feb/18 16:52
• Resolved: 19/Feb/18 16:49

Description

S3AInputStream handles any IOE through a close() of stream and single re-invocation of the read, with

• no backoff
• no abort of the HTTPS connection, which is just returned to the pool, If httpclient hasn't noticed the failure, it may get returned to the caller on the next read

Proposed

• switch to invoker
• retry policy explicitly for stream (EOF => throw, timeout => close, sleep, retry, etc)

We could think about extending the fault injection to inject stream read failures intermittently too, though it would need something in S3AInputStream to (optionally) wrap the http input streams with the failing stream.

Issue Links

• Is contained by: HADOOP-13761 S3Guard: implement retries for DDB failures and throttling; translate exceptions
• is related to: SPARK-23308 ignoreCorruptFiles should not ignore retryable IOException