[HADOOP-2178] Job history on HDFS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Hadoop Flags: Incompatible change

Release Note

This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" .
The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available.
The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs).
The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)

• Assignee: Amareshwari Sriramadasu
• Reporter: Amareshwari Sriramadasu
• Created: 09/Nov/07 09:55
• Updated: 08/Jul/09 16:52
• Resolved: 20/Feb/08 06:05

Description

This issue addresses the following items :

1. Check for accuracy of job tracker history logs.

2. After completion of the job, copy the JobHistory.log(Master index file) and the job history files to the DFS.

3. User can load the history with commands
bin/hadoop job -history <directory>
or
bin/hadoop job -history <jobid>
This will start a stand-alone jetty and load jsps

Issue Links

• is related to: HADOOP-1876 Persisting completed jobs status