[MAPREDUCE-7300] PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.3.0
• Component/s: mrv2
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 07/Oct/20 18:32
• Updated: 05/Jul/21 20:19
• Resolved: 05/Jul/21 20:19

Description

Related to MAPREDUCE-7282 a variant solution

1. we add a new method for committers to declare whether they can recover from a task attempt commit failure
2. default = true; v2 and (external) EMR spark committers can return false

execution engine - MR, Spark, can look at this after a task attempt fails to commit and decide what to do

recoverable: execute/commit another task attempt

non-recoverable, one of (Configured)

• warn and continue
• abort the job

with the job abort option, users would be confident that if a failure happened during the commit phase, they'd know about it and choose how to recover.

I'd use a long/unusual name, so that in, say, Spark, reflection to could be used to find and call the method & so compile against older releases

Issue Links

• is related to: SPARK-33019 Use spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=1 by default
• is superceded by: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• relates to: MAPREDUCE-7282 MR v2 commit algorithm should be deprecated and not the default