Issue Details (XML | Word | Printable)

Key: HADOOP-2119
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Amar Kamat
Reporter: Runping Qi
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

JobTracker becomes non-responsive if the task trackers finish task too fast

Created: 29/Oct/07 08:43 PM   Updated: 08/Jul/09 04:52 PM
Return to search
Component/s: None
Affects Version/s: 0.16.0
Fix Version/s: 0.17.0

Time Tracking:
Not Specified

File Attachments:
  File Name Date Attached Attached By Size
Text File Licensed for inclusion in ASF works HADOOP-2119-v4.1.patch 2008-03-13 06:08 PM Amar Kamat 52 kB
Text File Licensed for inclusion in ASF works HADOOP-2119-v5.1.patch 2008-03-20 05:58 PM Amar Kamat 55 kB
Text File Licensed for inclusion in ASF works HADOOP-2119-v5.1.patch 2008-03-20 04:33 AM Amar Kamat 52 kB
Text File Licensed for inclusion in ASF works HADOOP-2119-v5.2.patch 2008-03-21 10:04 PM Amar Kamat 51 kB
Text File Licensed for inclusion in ASF works hadoop-2119.patch 2007-11-29 02:24 AM Srikanth Kakani 11 kB
Text File hadoop-jobtracker-thread-dump.txt 2007-11-25 05:32 PM Christian Kunz 60 kB
Issue Links:
Blocker
 
Dependants
 
Reference

Hadoop Flags: Reviewed
Release Note:
This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.
Resolution Date: 26/Mar/08 10:18 PM


 Description  « Hide
I ran a job with 0 reducer on a cluster with 390 nodes.
The mappers ran very fast.
The jobtracker lacks behind on committing completed mapper tasks.
The number of running mappers displayed on web UI getting bigger and bigger.
The jos tracker eventually stopped responding to web UI.

No progress is reported afterwards.

Job tracker is running on a separate node.
The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Doug Cutting made changes - 29/Oct/07 09:52 PM
Field Original Value New Value
Fix Version/s 0.15.0 [ 12312565 ]
Fix Version/s 0.16.0 [ 12312740 ]
Description
I ran a job with 0 reducer on a cluster with 390 nodes.
The mappers ran very fast.
The jobtracker lacks behind on committing completed mapper tasks.
The number of running mappers displayed on web UI getting bigger and bigger.
The jos tracker eventually stopped responding to web UI.

No progress is reported afterwards.

Job tracker is running on a separate node.
The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).

 I ran a job with 0 reducer on a cluster with 390 nodes.
The mappers ran very fast.
The jobtracker lacks behind on committing completed mapper tasks.
The number of running mappers displayed on web UI getting bigger and bigger.
The jos tracker eventually stopped responding to web UI.

No progress is reported afterwards.

Job tracker is running on a separate node.
The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap space limit).
Christian Kunz made changes - 25/Nov/07 05:32 PM
Attachment hadoop-jobtracker-thread-dump.txt [ 12370161 ]
Srikanth Kakani made changes - 29/Nov/07 02:22 AM
Affects Version/s 0.16.0 [ 12312740 ]
Status Open [ 1 ] Patch Available [ 10002 ]
Srikanth Kakani made changes - 29/Nov/07 02:23 AM
Comment [ Patch that batches up promotions. ]
Srikanth Kakani made changes - 29/Nov/07 02:24 AM
Attachment hadoop-2119.patch [ 12370493 ]
Devaraj Das made changes - 29/Nov/07 09:12 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Christian Kunz made changes - 30/Nov/07 01:30 AM
Fix Version/s 0.16.0 [ 12312740 ]
Fix Version/s 0.15.2 [ 12312877 ]
Christian Kunz made changes - 30/Nov/07 01:35 AM
Priority Major [ 3 ] Blocker [ 1 ]
Christian Kunz made changes - 17/Dec/07 11:46 PM
Fix Version/s 0.15.2 [ 12312877 ]
Fix Version/s 0.16.0 [ 12312740 ]
Nigel Daley made changes - 23/Jan/08 12:57 AM
Fix Version/s 0.17.0 [ 12312913 ]
Fix Version/s 0.16.0 [ 12312740 ]
Priority Blocker [ 1 ] Critical [ 2 ]
Amar Kamat made changes - 28/Jan/08 09:28 AM
Link This issue depends on HADOOP-1985 [ HADOOP-1985 ]
Amar Kamat made changes - 30/Jan/08 03:57 PM
Assignee Amar Kamat [ amar_kamat ]
Owen O'Malley made changes - 07/Feb/08 02:52 AM
Link This issue is blocked by HADOOP-2790 [ HADOOP-2790 ]
Runping Qi made changes - 07/Feb/08 04:54 AM
Link This issue relates to HADOOP-2790 [ HADOOP-2790 ]
Runping Qi made changes - 07/Feb/08 04:56 AM
Link This issue is blocked by HADOOP-2790 [ HADOOP-2790 ]
Sameer Paranjpye made changes - 08/Feb/08 10:21 AM
Link This issue is related to HADOOP-2014 [ HADOOP-2014 ]
Amar Kamat made changes - 13/Feb/08 06:43 AM
Link This issue blocks HADOOP-2812 [ HADOOP-2812 ]
Amar Kamat made changes - 22/Feb/08 11:17 AM
Link This issue blocks HADOOP-2874 [ HADOOP-2874 ]
Amar Kamat made changes - 12/Mar/08 12:08 PM
Status Open [ 1 ] In Progress [ 3 ]
Amar Kamat made changes - 13/Mar/08 06:08 PM
Attachment HADOOP-2119-v4.1.patch [ 12377817 ]
Amar Kamat made changes - 17/Mar/08 04:36 AM
Status In Progress [ 3 ] Patch Available [ 10002 ]
Amar Kamat made changes - 20/Mar/08 04:33 AM
Attachment HADOOP-2119-v5.1.patch [ 12378295 ]
Amar Kamat made changes - 20/Mar/08 04:33 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Amar Kamat made changes - 20/Mar/08 04:33 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Owen O'Malley made changes - 20/Mar/08 05:37 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Amar Kamat made changes - 20/Mar/08 05:58 PM
Attachment HADOOP-2119-v5.1.patch [ 12378329 ]
Amar Kamat made changes - 21/Mar/08 10:04 PM
Attachment HADOOP-2119-v5.2.patch [ 12378413 ]
Amar Kamat made changes - 21/Mar/08 10:05 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Robert Chansler made changes - 25/Mar/08 03:03 AM
Fix Version/s 0.17.0 [ 12312913 ]
Devaraj Das made changes - 25/Mar/08 04:19 AM
Fix Version/s 0.17.0 [ 12312913 ]
Owen O'Malley made changes - 26/Mar/08 10:18 PM
Resolution Fixed [ 1 ]
Status Patch Available [ 10002 ] Resolved [ 5 ]
Owen O'Malley made changes - 26/Mar/08 10:19 PM
Comment [ I just committed this. ]
Devaraj Das made changes - 17/Apr/08 06:39 AM
Release Note This removes many inefficiencies in task placement and scheduling logic. The JobTracker would perform linear scans of the list of submitted tasks in cases where it did not find an obvious candidate task for a node. With better data structures for managing job state, all task placement operations now run in constant time (in most cases). Also, the task output promotions are batched.
Hadoop Flags [Reviewed]
Nigel Daley made changes - 21/May/08 08:05 PM
Status Resolved [ 5 ] Closed [ 6 ]
Owen O'Malley made changes - 08/Jul/09 04:52 PM
Component/s mapred [ 12310690 ]

Powered by a free Atlassian JIRA open source license for Apache Software Foundation. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.13.2-#335) - Bug/feature request - Atlassian news - Contact Administrators