[MAPREDUCE-972] distcp can timeout during rename operation to s3

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1
• Fix Version/s: 0.21.0
• Component/s: distcp, documentation
• Hadoop Flags: Reviewed
• Assignee: Aaron Kimball
• Reporter: Aaron Kimball
• Created: 11/Sep/09 02:34
• Updated: 24/Aug/10 22:17
• Resolved: 03/Nov/09 08:03

Description

rename() in S3 is implemented as copy + delete. The S3 copy operation can perform very slowly, which may cause task timeout.

Issue Links

• is blocked by: HADOOP-6324 FileSystem API should allow progress callbacks in rename and delete
• is related to: MAPREDUCE-1127 distcp should timeout later during S3-based transfers