[#HADOOP-4679] Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX

Hadoop Common

Datanode prints tons of log messages: Waiting for threadgroup to exit, active theads is XX

Created: 18/Nov/08 07:27 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.18.3

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
diskError.patch	2008-11-25 01:24 AM	Hairong Kuang	6 kB
diskError1.patch	2008-11-25 06:47 PM	Hairong Kuang	6 kB
diskError2.patch	2008-11-26 10:21 PM	Hairong Kuang	8 kB
diskError3-br18.patch	2008-12-03 08:02 PM	Hairong Kuang	7 kB
diskError3.patch	2008-12-02 09:59 PM	Hairong Kuang	9 kB

Issue Links:

Reference

This issue relates to:

HDFS-264 Better Datanode DiskOutOfSpaceException handling.

~~HADOOP-5114~~ A bunch of mapred unit tests are failing on Windows

This issue is related to:

~~HADOOP-4962~~ HADOOP-4679 to be fixed for branches >= 0.19

Hadoop Flags:	Reviewed
Release Note:	« Hide 1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; 2. Datanode checks disk in case of failure on creating a block file. Show » 1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; 2. Datanode checks disk in case of failure on creating a block file.
Resolution Date:	03/Dec/08 08:07 PM

Description

« Hide

When a data receiver thread sees a disk error, it immediately calls shutdown to shutdown DataNode. But the shutdown method does not return before all data receiver threads exit, which will never happen. Therefore the DataNode gets into a dead/live lock state, emitting tons of log messages: Waiting for threadgroup to exit, active threads is XX.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Hairong Kuang made changes - 18/Nov/08 07:52 PM

Field	Original Value	New Value
Component/s		dfs [ 12310710 ]

Hairong Kuang made changes - 25/Nov/08 01:24 AM

Attachment

diskError.patch [ 12394623 ]

Hairong Kuang made changes - 25/Nov/08 01:24 AM

Assignee

Hairong Kuang [ hairong ]

Tsz Wo (Nicholas), SZE made changes - 25/Nov/08 01:27 AM

Link

This issue relates to HADOOP-3574 [ HADOOP-3574 ]

Hairong Kuang made changes - 25/Nov/08 06:47 PM

Attachment

diskError1.patch [ 12394679 ]

Hairong Kuang made changes - 26/Nov/08 10:21 PM

Attachment

diskError2.patch [ 12394784 ]

Hairong Kuang made changes - 02/Dec/08 09:59 PM

Attachment

diskError3.patch [ 12395132 ]

Hairong Kuang made changes - 03/Dec/08 08:02 PM

Attachment

diskError3-br18.patch [ 12395207 ]

Hairong Kuang made changes - 03/Dec/08 08:07 PM

Hadoop Flags		[Reviewed]
Status	Open [ 1 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]
Fix Version/s		0.18.3 [ 12313494 ]
Release Note		1. Only datanode's offerService thread shutdown the datanode to avoid deadlock; 2. Datanode checks disk in case of failure on creating a block file.

Sean Knapp made changes - 31/Dec/08 03:58 AM

Link

This issue is related to ~~HADOOP-4962~~ [ ~~HADOOP-4962~~ ]

Raghu Angadi made changes - 26/Jan/09 11:16 PM

Link

This issue relates to ~~HADOOP-5114~~ [ ~~HADOOP-5114~~ ]

Nigel Daley made changes - 30/Jan/09 08:14 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

Owen O'Malley made changes - 08/Jul/09 04:43 PM

Component/s

dfs [ 12310710 ]