[HADOOP-1558] changes to OutputFormat to work on temporary directory to enable re-running crashed jobs (Issue: 1121)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Fix

Environment

all

• Assignee: Unassigned
• Reporter: Alejandro Abdelnur
• Created: 02/Jul/07 13:03
• Updated: 08/Jul/09 16:51
• Resolved: 13/Jan/08 06:14

Description

Add OutputFormat methods like:

/** Called to initialize output for this job. */
void initialize(JobConf job) throws IOException;

/** Called to finalize output for this job. */
void commit(JobConf job) throws IOException;

In the base implemenation for FileSystem output, initialize() might then create a temporary directory for the job, removing any that already exists, and commit could rename the temporary output directory to the final name.

The existing checkOutputSpecs() would continue to throw an exception if the final output already exists.

Issue Links

• blocks: HADOOP-1121 Recovering running/scheduled jobs after JobTracker failure
• is blocked by: HADOOP-1587 Tasks run by MiniMRCluster don't get sysprops from TestCases