[MAPREDUCE-157] Job History log file format is not friendly for external tools.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1
• Fix Version/s: 0.21.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Changes the Job History file format to use JSON.
Simplifies the Job History Parsing logic
Removes duplication of code between HistoryViewer and the JSP files
History Files are now named as JobID_user
Introduces a new cluster level configuration "mapreduce.cluster.jobhistory.maxage" for configuring the amount of time history files are kept before getting cleaned up
The configuration "hadoop.job.history.user.location" is no longer supported.

• Assignee: Jothi Padmanabhan
• Reporter: Owen O'Malley
• Created: 14/May/09 16:38
• Updated: 12/Jan/11 04:45
• Resolved: 17/Sep/09 05:10

Description

Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

Issue Links

• blocks: MAPREDUCE-864 Enhance JobClient API implementations to look at history files to get information about jobs that are not in memory
• blocks: MAPREDUCE-975 Add an API in job client to get the history file url for a given job id
• blocks: MAPREDUCE-198 Log job history events to a common dump file
• blocks: MAPREDUCE-980 Modify JobHistory to use Avro for serialization instead of raw JSON
• blocks: MAPREDUCE-277 Job history counters should be avaible on the UI.
• incorporates: MAPREDUCE-881 Jobtracker continues even if History initialization fails
• incorporates: MAPREDUCE-926 History viewer on web UI should filter by job-id also
• relates to: MAPREDUCE-2251 Remove mapreduce.job.userhistorylocation config