[HADOOP-18012] ABFS: Enable config controlled ETag check for Rename idempotency

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.4.0, 3.3.6
• Component/s: fs/azure
• Labels: pull-request-available
• Target Version/s: 3.3.9
• Assignee: Sree Bhattacharyya
• Reporter: Sneha Vijayarajan
• Created: 15/Nov/21 17:39
• Updated: 26/Feb/24 13:34
• Resolved: 06/Apr/23 14:55

Description

To support recovery of comms failure during rename, the abfs client fetches the etag of the source file, and when recovering from a failure uses this tag to determine whether the rename succeeded before the failure happened

The relevant configuration option is fs.azure.enable.rename.resilience; default value is: true

1. This works for files, but not directories
2. this adds the overhead of a HEAD request before each rename.
3. the option can be disabled by setting "fs.azure.enable.rename.resilience" to false

Note: the manifest committer collects etags during task commit and supplies them to the abfs client for the rename, which avoids the need for a HEAD call.

Issue Links

• is duplicated by: HADOOP-18667 ABFS: eTag Check to mitigate Rename Failures
• is duplicated by: HADOOP-18425 [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop
• relates to: HADOOP-17934 NullPointerException when no HTTP response set on AbfsRestOperation
• relates to: HADOOP-18002 abfs rename idempotency broken -remove recovery
• relates to: HADOOP-17979 Interface EtagSource to allow FileStatus subclasses to provide etags
• relates to: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• links to: GitHub Pull Request #5488