[HADOOP-997] Implement S3 retry mechanism for failed block transfers

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.12.0
• Component/s: fs
• Assignee: Thomas White
• Reporter: Thomas White
• Created: 08/Feb/07 21:34
• Updated: 02/Mar/07 23:02
• Resolved: 21/Feb/07 22:03

Description

HADOOP-882 improves S3FileSystem so that when certain communications problems with S3 occur the operation is retried. However, the retry mechanism cannot handle a block transfer failure, since blocks may be very large and we don't want to buffer them in memory. This improvement is to write a wrapper (using java.lang.reflect.Proxy if possible - see discussion in HADOOP-882) that can retry block transfers.

Issue Links

• is related to: HADOOP-882 S3FileSystem should retry if there is a communication problem with S3