[MAPREDUCE-3567] Extraneous JobConf objects in AM heap

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Component/s: mr-am, mrv2, performance
• Target Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Vinod Kumar Vavilapalli
• Created: 15/Dec/11 21:44
• Updated: 05/Mar/12 02:49
• Resolved: 22/Dec/11 22:36

Description

MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and TaskImpl which occupy non-trivial amount of heap.

While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing those extraneous objects helped in addressing OOM with 2GB AM heap size.