[HADOOP-815] Investigate and fix the extremely large memory-footprint of JobTracker

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.1
• Fix Version/s: 0.10.1
• Assignee: Arun Murthy
• Reporter: Arun Murthy
• Created: 12/Dec/06 06:19
• Updated: 02/May/13 02:29
• Resolved: 08/Jan/07 19:25

Description

The JobTracker's memory footprint seems excessively large, especially when many jobs are submitted.

Here is the 'top' output of a JobTracker which has scheduled ~1k jobs thus far:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31877 arunc 19 0 2362m 261m 13m S 14.0 12.9 24:48.08 java

Clearly VIRTual memory of 2364Mb v/s 261Mb of RESident memory is symptomatic of this issue...

Issue Links

• incorporates: HADOOP-740 JobTracker does not clean up task entries at job completion
• incorporates: HADOOP-787 job tracker should limit the number of completed/failed jobs in RAM
• is depended upon by: HADOOP-600 Race condition in JobTracker updating the task tracker's status while declaring it lost