[MAPREDUCE-416] Move the completed jobs' history files to a DONE subdirectory inside the configured history directory

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.21.0

Release Note

Once the job is done, the history file and associated conf file is moved to history.folder/done folder. This is done to avoid garbling the running jobs' folder and the framework no longer gets affected with the files in the done folder. This helps in 2 was
1) ls on running folder (recovery) is faster with less files
2) changes in running folder results into FileNotFoundException.


So with existing code, the best way to keep the running folder clean is to note the id's of running job and then move files that are not in this list to the done folder. Note that on an avg there will be 2 files in the history folder namely
1) job history file
2) conf file.

With restart, there might be more than 2 files, mostly the extra conf files. In such a case keep the oldest conf file (based on timestamp) and delete the rest. Note that this its better to do this when the jobtracker is down.

• Assignee: Amar Kamat
• Reporter: Devaraj Das
• Created: 09/Jun/09 05:43
• Updated: 24/Aug/10 21:13
• Resolved: 29/Jun/09 13:23

Description

Whenever a job completes, the history file can be moved to a directory called DONE. That would make the management of job history files easier (for example, administrators can move the history files from that directory to some other place, delete them, archive them, etc.).

Issue Links

• incorporates: MAPREDUCE-276 Multiple copies of jobconf is present in history-dir after restart