NameNode should report status and performance for each replica of image and log

• Key: HADOOP-4029
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Boris Shkolnik
• Reporter: Robert Chansler
• Affects Version/s: 0.20.0
• Fix Version/s: 0.20.0

Description

The administrator should have immediate access to the status of each replica. In addition to reporting whether the replica is active or not, measured performance should be reported so as to indicate whether a replica might impact cluster performance or might be in danger of failing.

We're probably not close to doing automated replica fail over, but we can give the administrator some clues.

Comments

[Boris Shkolnik]

This patch include the following changes:
1. additional storage directory (in Storage class) to keep removed(failed) storages and accessors to this list
2. changes to FSImage to add storages to this list (when IO failes)
3. changes to the dfshealth.jsp page to display both lists (valid and failed storages)
4. List of live/dead nodes is moved to a separate page.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394796/HADOOP-4029.patch
against trunk revision 721415.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/console

This message is automatically generated.

[Boris Shkolnik]

-1 findbugs
These warnings should be ignored. See JIRA HADOOP-4755

-1 core tests
Both failed tests are in mapred and not related to the changes in the patch.

[Konstantin Shvachko]

1. Storage.removedStorageDirs should be a member of FSImage rather than Storage class.
2. In FSImage.java redundant import of StorageDirectory.
3. Remove comments with test printouts from FSImage.processIOError();
4. Replace tabs with spaces in FSImage.processIOError();
5. In dfshealth.jsp
   • Instead of "FSImage Storage Configuration:" I'd rather say "NameNode Storage:"
   • Column "Dir" should rather be named "Storage Directory".

[Boris Shkolnik]

implemented suggestions given by Konstantin

[Konstantin Shvachko]

Minor comments:

1. FSImage.getStorageDirs() is redundant. You should use Storage.getNumStorageDirs() and Storage.getStorageDir(idx) instead.
2. Could you please remove empty lines after it.remove() in FSImage.
3. Unnecessary empty lines changes in dfshealth.jsp.

[Boris Shkolnik]

implemented Konstantin's comments

[Konstantin Shvachko]

+1

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12395466/HADOOP-4029-2.patch
against trunk revision 724229.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

I just committed this. Thanks, Boris!

[Hudson]

Integrated in Hadoop-trunk #683 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/683/)
. Add NameNode storage information to the dfshealth page and move DataNode information to a separated page. (Boris Shkolnik via szetszwo)

[Robert Chansler]

Edit release note for publication.