[MAPREDUCE-863] Improve/standardize job history file content format and the management of the history files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: jobtracker
• Assignee: Unassigned
• Reporter: Devaraj Das
• Created: 13/Aug/09 14:55
• Updated: 23/Jul/14 21:03
• Resolved: 23/Jul/14 21:03

Description

This jira will track the various improvements that will be done in the JobHistory component of Hadoop. The main goals:
1) Make the content format friendly for consumption by other apps
2) Improve the management of the history files.
3) Be able to completely rely on the JobHistory files for anything to do with getting the status of completed jobs, both at the JobClient end, and at the web-UI end

Sub-Tasks

• Job History log file format is not friendly for external tools.: Closed
• Move completed Job history files to HDFS: Closed
• Add a cache for retired jobs with minimal job info and provide a way to access history file url: Closed
• Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory: Resolved
• Clean up the job Retire code: Closed
• Add an API in job client to get the history file url for a given job id: Closed