[HADOOP-13823] s3a rename: fail if dest file exists

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 2.8.0, 3.0.0-alpha2
• Component/s: fs/s3
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 17/Nov/16 10:13
• Updated: 04/Feb/17 16:29
• Resolved: 29/Nov/16 00:46

Description

HIVE-15199 shows that s3a allows rename onto an existing file, which is something HDFS, azure and s3n do not permit (though file:// does). This is breaking bits of Hive, is an inconsistency with HDFS and a regression compared to s3n semantics.

I propose: rejecting the rename on a file -> file rename if the destination exists (easy) and changing the s3a.xml contract file to declare the behavior change; this is needed for AbstractContractRenameTest.testRenameFileOverExistingFile to handle the changed semantics.

Issue Links

• is depended upon by: HADOOP-13600 S3a rename() to copy files in a directory in parallel
• is related to: HIVE-15199 INSERT INTO data on S3 is replacing the old rows with the new ones