[MAPREDUCE-3512] Batch jobHistory disk flushes

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Component/s: mr-am, mrv2
• Hadoop Flags: Reviewed
• Release Note: Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM.
• Assignee: Siddharth Seth
• Reporter: Siddharth Seth
• Created: 06/Dec/11 00:33
• Updated: 05/Mar/12 02:49
• Resolved: 12/Jan/12 01:44

Description

The mr-am flushes each individual job history event to disk for AM recovery. The history even handler ends up with a significant backlog for tests like MAPREDUCE-3402.
History events could be batched up based on num records / time / TaskFinishedEvents to reduce the number of DFS writes - with the potential drawback of having to rerun some tasks during AM recovery.