[#HADOOP-3864] JobTracker lockup due to JobInProgress.initTasks taking significant time for large jobs on large clusters

Hadoop Common

JobTracker lockup due to JobInProgress.initTasks taking significant time for large jobs on large clusters

Created: 30/Jul/08 06:06 PM Updated: 08/Jul/09 04:52 PM

Return to search

Component/s:

None

Affects Version/s:

0.18.0

Fix Version/s:

0.19.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	HADOOP-3864_0_20080830.patch	2008-07-31 02:15 AM	Arun C Murthy	4 kB

Hadoop Flags:	Reviewed
Resolution Date:	08/Aug/08 11:42 PM

Description

« Hide

JobInProgress.initTasks takes significant amount of time on a large cluster for large jobs (55k maps * 3 splits), during which the JobInProgress object is locked up.

Simultaneously the JobClient is calling JobTracker.getTaskCompletionEvents which locks the JobTracker & tries to lock the JobInProgress, there-by it starves all heartbeats which are trying to lock the JobTracker - resulting in a lockup.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

No work has yet been logged on this issue.