[MAPREDUCE-7474] [ABFS] Improve commit resilience and performance in Manifest Committer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.6
• Fix Version/s: 3.3.9, 3.5.0, 3.4.1
• Component/s: client
• Labels: pull-request-available
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 03/Apr/24 15:42
• Updated: 15/May/24 15:25
• Resolved: 15/May/24 15:25

Description

• Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled.
• large burst of delete calls noted: are they needed?

relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only.

Initial proposed changes

• retry recovery on task commit rename, always (repeat save, delete, rename)
• audit delete use and see if it can be pruned

Issue Links

• is depended upon by: HADOOP-19092 ABFS phase 4: post Hadoop 3.4.0 features
• relates to: HADOOP-19093 Improve rate limiting through ABFS in Manifest Committer
• links to: GitHub Pull Request #6716
• links to: GitHub Pull Request #6824
• links to: GitHub Pull Request #6825