[#HADOOP-5009] DataNode#shutdown sometimes leaves data block scanner verification log unclosed

Hadoop Common

DataNode#shutdown sometimes leaves data block scanner verification log unclosed

Created: 09/Jan/09 11:57 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

0.17.0

Fix Version/s:

0.19.1

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
blockScannerFd.patch	2009-01-10 12:12 AM	Hairong Kuang	2 kB
blockScannerFd1.patch	2009-01-20 10:15 PM	Hairong Kuang	2 kB

Hadoop Flags:	Reviewed
Resolution Date:	21/Jan/09 09:59 PM

Description

« Hide

When datanode gets shutdown by calling DataNode#shutdown, it occasionally leaves the data block scanner verification log unclosed. There are two possible causes:
1. DataNode does not wait until block scanner thread to exit.
2. DataBlockScanner does not guarantee that the verification log is closed if the scanner is interrupted.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Hairong Kuang added a comment - 10/Jan/09 12:12 AM

The attached patch attempts to fix the problem.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 12/Jan/09 10:34 PM

+1 Patch looks good. Two minor suggestions:

using blockScannerThread.join() may be dangerous if blockScannerThread does not terminate. How about replace join() with blockScannerThread.join(millis), so that it will time out?
I think it is better to use try-finally in closing the readers. See a comment in HADOOP-2926.

[ Permlink | « Hide ]

Hairong Kuang added a comment - 20/Jan/09 10:15 PM

This patch incorporates Nicholas' comment.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 21/Jan/09 12:27 AM

+1 the new patch looks good.

[ Permlink | « Hide ]

Hairong Kuang added a comment - 21/Jan/09 09:43 PM

[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

ant test-core fails only TestMapReduceLocal which is a known issue.

[ Permlink | « Hide ]

Hairong Kuang added a comment - 21/Jan/09 09:59 PM

I've just resolved this.