[HADOOP-16190] S3A copyFile operation to include source versionID or etag in the copy request

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.9.2, 3.0.3
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 14/Mar/19 14:48
• Updated: 17/May/19 11:07
• Resolved: 17/May/19 11:05

Description

The S3 CopyObjectRequest structure used to build a copy request takes, optionally

• a version ID of the source file
• the etag of the source file

using these will guarantee that the source file is copied exactly once

1. When working with S3Guard in auth mode, and the etag/version are kept in the DDB table, that should be the authoritative source of metadata: fail fast if the file is out of date
2. And when non-auth mode is used, the getObjectMetadata check made should update the s3guard table if needed.

Issue Links

• is depended upon by: HADOOP-16085 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite
• is related to: HADOOP-16189 S3A copy/rename of large files to be parallelized as a multipart operation
• is superceded by: HADOOP-16085 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite
• links to: GitHub Pull Request #606