[#HADOOP-2119] JobTracker becomes non-responsive if the task trackers finish task too fast

Hadoop Common

JobTracker becomes non-responsive if the task trackers finish task too fast

Created: 29/Oct/07 08:43 PM Updated: 08/Jul/09 04:52 PM

Return to search

Component/s:

None

Affects Version/s:

0.16.0

Fix Version/s:

0.17.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-2119-v4.1.patch	2008-03-13 06:08 PM	Amar Kamat	52 kB
HADOOP-2119-v5.1.patch	2008-03-20 05:58 PM	Amar Kamat	55 kB
HADOOP-2119-v5.1.patch	2008-03-20 04:33 AM	Amar Kamat	52 kB
HADOOP-2119-v5.2.patch	2008-03-21 10:04 PM	Amar Kamat	51 kB
hadoop-2119.patch	2007-11-29 02:24 AM	Srikanth Kakani	11 kB
hadoop-jobtracker-thread-dump.txt	2007-11-25 05:32 PM	Christian Kunz	60 kB

Issue Links:

Blocker

This issue blocks:

MAPREDUCE-262 Optimize finding of speculative tasks

MAPREDUCE-315 Bias the decision of task scheduling (both for not-running and running) on node metrics (load, processing rate etc).

Dependants

This issue depends on:

~~HADOOP-1985~~ Abstract node to switch mapping into a topology service class used by namenode and jobtracker

Reference

This issue relates to:

~~HADOOP-2790~~ TaskInProgress.hasSpeculativeTask is very inefficient

This issue is related to:

MAPREDUCE-93 Job Tracker should prefer input-splits from overloaded racks

Hadoop Flags:	Reviewed
Release Note:	« Hide This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched. Show » This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.
Resolution Date:	26/Mar/08 10:18 PM

Description

« Hide

I ran a job with 0 reducer on a cluster with 390 nodes.
The mappers ran very fast.
The jobtracker lacks behind on committing completed mapper tasks.
The number of running mappers displayed on web UI getting bigger and bigger.
The jos tracker eventually stopped responding to web UI.

No progress is reported afterwards.

Job tracker is running on a separate node.
The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

No work has yet been logged on this issue.