[MAPREDUCE-5485] Allow repeating job commit by extending OutputCommitter API

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Target Version/s: 2.7.3
• Hadoop Flags: Reviewed

Release Note

Previously, the MR job will get failed if AM get restarted for some reason (like node failure, etc.) during its doing commit job no matter if AM attempts reach to the maximum attempts.
In this improvement, we add a new API isCommitJobRepeatable() to OutputCommitter interface which to indicate if job's committer can do commitJob again if previous commit work is interrupted by NM/AM failures, etc. The instance of OutputCommitter, which support repeatable job commit (like FileOutputCommitter in algorithm 2), can allow AM to continue the commitJob() after AM restart as a new attempt.

• Assignee: Junping Du
• Reporter: Nemon Lou
• Created: 28/Aug/13 13:07
• Updated: 25/Oct/19 20:27
• Resolved: 17/Nov/15 01:14

Description

There are chances MRAppMaster crush during job committing,or NodeManager restart cause the committing AM exit due to container expire.In these cases ,the job will fail.
However,some jobs can redo commit so failing the job becomes unnecessary.
Let clients tell AM to allow redo commit or not is a better choice.
This idea comes from Jason Lowe's comments in MAPREDUCE-4819

Issue Links

• breaks: MAPREDUCE-6555 TestMRAppMaster fails on trunk
• breaks: MAPREDUCE-6595 Fix findbugs warnings in OutputCommitter and FileOutputCommitter
• is depended upon by: MAPREDUCE-6608 Work Preserving AM Restart for MapReduce
• is duplicated by: MAPREDUCE-6437 Add retry on some connection exception on job commit phase
• is related to: MAPREDUCE-6545 Test committer.commitJob() behavior during committing when MR AM get failed.
• relates to: MAPREDUCE-6478 Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob().
• relates to: MAPREDUCE-4815 Speed up FileOutputCommitter#commitJob for many output files