[MAPREDUCE-7432] Make Manifest Committer the default for abfs and gcs

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: client
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Release Note: By default, the mapreduce manifest committer is used for jobs working with abfs and gcs.. Hadoop mapreduce jobs will pick this up automatically; for Spark it is a bit complicated: read the docs to see the steps required.
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 09/Feb/23 14:37
• Updated: 16/Jan/24 08:42
• Resolved: 27/Jun/23 12:57

Description

Switch to the manifest committer as default for abfs and gcs

• abfs: needed for performance, scale and resilience under some failure modes
• gcs: provides correctness through atomic task commit and better job commit performance

Issue Links

• links to: GitHub Pull Request #5378