[#HADOOP-4103] Alert for missing blocks

Hadoop Common

Alert for missing blocks

Created: 07/Sep/08 07:18 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

0.17.2

Fix Version/s:

0.20.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-4103-branch-20.patch	2009-03-04 03:17 AM	Raghu Angadi	18 kB
HADOOP-4103.patch	2009-03-02 10:03 PM	Raghu Angadi	22 kB
HADOOP-4103.patch	2009-02-27 01:00 AM	Raghu Angadi	22 kB
HADOOP-4103.patch	2009-02-26 01:24 AM	Raghu Angadi	23 kB
HADOOP-4103.patch	2009-02-20 01:35 AM	Raghu Angadi	19 kB

Hadoop Flags:	Incompatible change, Reviewed
Release Note:	Modified dfsadmin -report to report under replicated blocks. blocks with corrupt replicas, and missing blocks".
Resolution Date:	04/Mar/09 03:18 AM

Description

« Hide

A whole bunch of datanodes became dead because of some network problems resulting in heartbeat timeouts although datanodes were fine.

Many processes started to fail because of the corrupted filesystem.

In order to catch and diagnose such problems faster the namenode should detect the corruption automatically and provide a way to alert operations. At the minimum it should show the fact of corruption on the GUI.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Raghu Angadi added a comment - 25/Nov/08 09:40 PM

I thinking of implementing a background fsck on NameNode. This will share/reuse most of the code with current Fsck. The extra features will be to facilitate an admin to quickly check if there something odd (e.g. ability list last 100 or so blocks in inconsistent state).

Based on this background check there could be further improvements to monitoring more alarms over time.. as well as reducing latency of detection.

This feature will be optional. Scan period could be around a day.

[ Permlink | « Hide ]

Raghu Angadi added a comment - 24/Jan/09 01:20 AM - edited

(Edit : formatting only)

The scope of the fix is narrowed to the following :

NameNode webui shows in (probably in red) indicating if there are any missing blocks.
- will mostly add simon stats for such a number.

'dfsadmin -metasave' can be used to find all the missing blocks
- a later jira will enhance -metasave or have different command that is more user friendly. currently -metasave is mainly meant for developers.

For this to be a straight forward fix, I need to make one policy change: currently if a block does not have any good replicas left it is not included in "neededReplications" list. I think this was done mainly as an "optimization". But a cluster should not have any blocks this state. even 'neededReplications' name implies such blocks should be included. It would be better if I don't need to add another list that need to be maintained.

[ Permlink | « Hide ]

Raghu Angadi added a comment - 20/Feb/09 01:35 AM

The patch for missing block alerts. A user can monitor this in multiple ways :

'bin/hdfs dfsadmin -report' reports this count.
A warning is pasted in red on NameNode front page
new stat is added (for Simon, for e.g.).
- Also added a stat to report size of corrupt replicas map

Once the alert is noticed, admin can run 'dfsadmin -metasave' to find out which specific blocks are missing. 'metasave' is improved a bit to list replica info for each block in 'neededReplication' list and the line for a missing blocks contains the word "MISSING".

This is a very non-intrusive change, thus fairly safe for backporting. No new state or data structures for NN to track.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 25/Feb/09 02:06 AM

1. NamenodeProtocol.getStats() method documentation needs to be updated about the fourth stat that is being reported
2. DFSAdmin.java - remove space before : in "Missing Blocks (approx) : ". Additionally is it a good idea to print number of corrupt blocks, pending replication, scheduled replication and under replicated block counts in the report? Currently what is printed in dfsadmin report is also printed in the cluster summary on namenode web page. It may be a good idea to keep both of them consistent.
3. FSNamesystem.java computeReplicationWork() move the added code block that sets missingBlocksInCurIter, missingBlocksInPrevIter to zero, above the comments preceding it.

Would this change be incompatible because of change in the output of dfsadmin report command?

[ Permlink | « Hide ]

Raghu Angadi added a comment - 26/Feb/09 01:24 AM

Thanks Suresh.

Updated patch includes all the suggestions.

'dfsadmin -report' now prints 3 extra lines one for each of "Under replicated blocks" "Blocks with corrupt replicas" "Missing blocks". The last two counts should be zero normally. The first count should be low and should keep going down.

Regd whether it should be treated as "imcompatible" change.. I personally don't think so. But does not matter either way.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 26/Feb/09 09:50 PM

Comments:

DFSAdmin.java please remove the space before : in the newly introduced output
NameNodeMetrics.numBlocksCorrupted exposes the same data as FSNamesystemMetrics.corruptReplicaBlocks. Not sure where the new metrics introduced by this patch should go into

[ Permlink | « Hide ]

Raghu Angadi added a comment - 27/Feb/09 01:00 AM

Thanks Suresh.

Attached patch fixes both. The new stat for corrupt block is not required since it is already there. I didn't see that earlier.

[ Permlink | « Hide ]

Suresh Srinivas added a comment - 27/Feb/09 02:21 AM

+1 for the patch

[ Permlink | « Hide ]

Raghu Angadi added a comment - 27/Feb/09 07:13 PM

I hope this gets marked for 0.20. It is pretty safe. Otherwise , I am pretty sure I will have to back port it again in near future and duplicate considerable constant effort associated with a new jira and a commit.