[MAPREDUCE-4912] Investigate ways to clean up double job commit prevention

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: mrv2
• Assignee: Unassigned
• Reporter: Robert Joseph Evans
• Created: 04/Jan/13 15:28
• Updated: 04/Jan/13 15:28

Description

Once MAPREDUCE-4819 goes in it fixes the issue where an OutputCommiter can double commit a job. So that the output will never be touched after the job informs externally of success or failure.

The code and design could potentially use some cleanup and refactoring.

Issues brought up that should be investigated include:

1. reporting KILL for killed jobs if they crash after the kill happens instead of error.
2. using the job history log for recording the commit status instead of separate external files in HDFS.
3. Placing the recovery/retry logic in the commit handler instead of the MRAppMaster, and having the recovery service replay the logs as it normally does for recovery.

This is not meant to be things that must be done, but alternatives that might clean up the code.