[HADOOP-17979] Interface EtagSource to allow FileStatus subclasses to provide etags

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.3.2
• Component/s: fs, fs/azure, fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 26/Oct/21 16:04
• Updated: 23/Mar/23 11:36
• Resolved: 29/Nov/21 16:22

Description

Various objects stores provide etags in their FileStatus implementations

Make these values accessible

• new interface EtagFromFileStatus to be implemented when provided
• filesystem.md to declare requirements of etags (constant between LIST and HEAD)...
• path capabilities for (a) etag and (b) etags consistent across rename

Add implementation for abfs, later s3a (and google gcs)

This is initially to handle recovery from certain failures in job commit against abfs, but it would allow a cloud-ready version of distcp to track etags of uploaded files, so diff properly.

Issue Links

• blocks: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• causes: HADOOP-18075 ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException
• incorporates: HADOOP-17492 abfs listLocatedStatus to support incremental/async page fetching
• is related to: HADOOP-18012 ABFS: Enable config controlled ETag check for Rename idempotency
• is related to: HADOOP-18425 [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop
• is related to: HADOOP-17981 Support etag-assisted renames in FileOutputCommitter
• is related to: HADOOP-18672 ask: abfs connector to support checksum
• relates to: HADOOP-18002 abfs rename idempotency broken -remove recovery
• links to: GitHub Pull Request #3633