[MAPREDUCE-4819] AM can rerun job after reporting final job status to the client

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.1-alpha
• Fix Version/s: 2.0.3-alpha, 0.23.6
• Component/s: mr-am
• Assignee: Bikas Saha
• Reporter: Jason Darrell Lowe
• Created: 26/Nov/12 19:03
• Updated: 03/Sep/14 23:25
• Resolved: 04/Jan/13 20:44

Description

If the AM reports final job status to the client but then crashes before unregistering with the RM then the RM can run another AM attempt. Currently AM re-attempts assume that the previous attempts did not reach a final job state, and that causes the job to rerun (from scratch, if the output format doesn't support recovery).

Re-running the job when we've already told the client the final status of the job is bad for a number of reasons. If the job failed, it's confusing at best since the client was already told the job failed but the subsequent attempt could succeed. If the job succeeded there could be data loss, as a subsequent job launched by the client tries to consume the job's output as input just as the re-attempt starts removing output files in preparation for the output commit.

Issue Links

• is duplicated by: YARN-243 Job Client doesn't give progress for Application Master Retries
• is related to: MAPREDUCE-4831 Task commit can occur more than once due to AM retries
• is related to: MAPREDUCE-4813 AM timing out during job commit
• is related to: MAPREDUCE-4832 MR AM can get in a split brain situation
• is related to: MAPREDUCE-4913 TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits
• relates to: MAPREDUCE-5476 Job can fail when RM restarts after staging dir is cleaned but before MR successfully unregister with RM