[MAPREDUCE-4737] Hadoop does not close output file / does not call Mapper.cleanup if exception in map

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1-win, 2.0.3-alpha, 1.1.2
• Fix Version/s: 1.2.0, 1-win, 2.1.0-beta
• Target Version/s: 1.2.0, 1-win, 2.1.0-beta
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour.

• Assignee: Arun Murthy
• Reporter: Daniel Dai
• Created: 09/Oct/12 07:45
• Updated: 15/May/13 05:16
• Resolved: 24/Apr/13 17:44

Description

Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.