[HADOOP-17981] Support etag-assisted renames in FileOutputCommitter

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.4.0
• Component/s: fs, fs/azure
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 28/Oct/21 15:24
• Updated: 26/Aug/22 14:01
• Resolved: 10/Nov/21 16:00

Description

To deal with some throttling/retry issues in object stores,
pass the FileStatus entries retrieved during listing
into a private interface ResilientCommitByRename which filesystems
may implement to use extra attributes in the listing (etag, version)
to constrain and validate the operation.

Although targeting azure, GCS and others could use. no point in S3A as they shouldn't use this committer.

1. And we are not going to do any changes to FileSystem as there are explicit guarantees of public use and stability.
   I am not going to make that change as the hive thing that will suddenly start expecting it to work forever.
2. I'm not planning to merge this in, as the manifest committer is going to include this and more (MAPREDUCE-7341)

However, I do need to get this in on a branch, so am doing this work on trunk for dev & test and for others to review

Issue Links

• is related to: HADOOP-18002 abfs rename idempotency broken -remove recovery
• is related to: HADOOP-18425 [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop
• is related to: HDDS-2935 Support for getFileChecksum in OzoneFS
• relates to: HADOOP-17979 Interface EtagSource to allow FileStatus subclasses to provide etags
• relates to: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• links to: GitHub Pull Request #3597
• links to: GitHub Pull Request #3611