[MAPREDUCE-1316] JobTracker holds stale references to retired jobs via unreported tasks

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.21.0
• Component/s: jobtracker
• Hadoop Flags: Reviewed

Release Note

JobTracker holds stale references to TaskInProgress objects and hence indirectly holds reference to retired jobs resulting into memory leak. Only task-attempts which are yet to report their status are left behind in the memory. All the task-attempts are now removed from the JobTracker by iterating over the scheduled task-attempt ids instead of iterating over available task statuses.

• Assignee: Amar Kamat
• Reporter: Amar Kamat
• Created: 18/Dec/09 09:56
• Updated: 24/Aug/10 21:19
• Resolved: 19/Jan/10 15:33

Description

JobTracker fails to remove unreported tasks' mapping from taskToTIPMap if the job finishes and retires. Unreported tasks refers to tasks that were scheduled but the tasktracker did not report back with the task status. In such cases a stale reference is held to TaskInProgress (and thus JobInProgress) long after the job is gone leading to memory leak.

Issue Links

• is depended upon by: MAPREDUCE-331 Make jobtracker resilient to memory issues
• is related to: MAPREDUCE-1387 Incorrect synchronization of {map|reduce|cleanup|setup} tasks in JobInProgress
• is related to: MAPREDUCE-1386 'ant javadoc' fails