Hadoop Common
  1. Hadoop Common
  2. HADOOP-4029

NameNode should report status and performance for each replica of image and log

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Added name node storage information to the dfshealth page, and moved data node information to a separated page.

      Description

      The administrator should have immediate access to the status of each replica. In addition to reporting whether the replica is active or not, measured performance should be reported so as to indicate whether a replica might impact cluster performance or might be in danger of failing.

      We're probably not close to doing automated replica fail over, but we can give the administrator some clues.

      1. HADOOP-4029.patch
        19 kB
        Boris Shkolnik
      2. HADOOP-4029-1.patch
        18 kB
        Boris Shkolnik
      3. HADOOP-4029-2.patch
        18 kB
        Boris Shkolnik

        Issue Links

          Activity

          Hide
          Boris Shkolnik added a comment -

          This patch include the following changes:
          1. additional storage directory (in Storage class) to keep removed(failed) storages and accessors to this list
          2. changes to FSImage to add storages to this list (when IO failes)
          3. changes to the dfshealth.jsp page to display both lists (valid and failed storages)
          4. List of live/dead nodes is moved to a separate page.

          Show
          Boris Shkolnik added a comment - This patch include the following changes: 1. additional storage directory (in Storage class) to keep removed(failed) storages and accessors to this list 2. changes to FSImage to add storages to this list (when IO failes) 3. changes to the dfshealth.jsp page to display both lists (valid and failed storages) 4. List of live/dead nodes is moved to a separate page.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12394796/HADOOP-4029.patch
          against trunk revision 721415.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12394796/HADOOP-4029.patch against trunk revision 721415. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3670/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          -1 findbugs
          These warnings should be ignored. See JIRA HADOOP-4755

          -1 core tests
          Both failed tests are in mapred and not related to the changes in the patch.

          Show
          Boris Shkolnik added a comment - -1 findbugs These warnings should be ignored. See JIRA HADOOP-4755 -1 core tests Both failed tests are in mapred and not related to the changes in the patch.
          Hide
          Konstantin Shvachko added a comment -
          1. Storage.removedStorageDirs should be a member of FSImage rather than Storage class.
          2. In FSImage.java redundant import of StorageDirectory.
          3. Remove comments with test printouts from FSImage.processIOError();
          4. Replace tabs with spaces in FSImage.processIOError();
          5. In dfshealth.jsp
            • Instead of "FSImage Storage Configuration:" I'd rather say "NameNode Storage:"
            • Column "Dir" should rather be named "Storage Directory".
          Show
          Konstantin Shvachko added a comment - Storage.removedStorageDirs should be a member of FSImage rather than Storage class. In FSImage.java redundant import of StorageDirectory. Remove comments with test printouts from FSImage.processIOError(); Replace tabs with spaces in FSImage.processIOError(); In dfshealth.jsp Instead of "FSImage Storage Configuration:" I'd rather say "NameNode Storage:" Column "Dir" should rather be named "Storage Directory".
          Hide
          Boris Shkolnik added a comment -

          implemented suggestions given by Konstantin

          Show
          Boris Shkolnik added a comment - implemented suggestions given by Konstantin
          Hide
          Konstantin Shvachko added a comment -

          Minor comments:

          1. FSImage.getStorageDirs() is redundant. You should use Storage.getNumStorageDirs() and Storage.getStorageDir(idx) instead.
          2. Could you please remove empty lines after it.remove() in FSImage.
          3. Unnecessary empty lines changes in dfshealth.jsp.
          Show
          Konstantin Shvachko added a comment - Minor comments: FSImage.getStorageDirs() is redundant. You should use Storage.getNumStorageDirs() and Storage.getStorageDir(idx) instead. Could you please remove empty lines after it.remove() in FSImage. Unnecessary empty lines changes in dfshealth.jsp.
          Hide
          Boris Shkolnik added a comment -

          implemented Konstantin's comments

          Show
          Boris Shkolnik added a comment - implemented Konstantin's comments
          Hide
          Konstantin Shvachko added a comment -

          +1

          Show
          Konstantin Shvachko added a comment - +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12395466/HADOOP-4029-2.patch
          against trunk revision 724229.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12395466/HADOOP-4029-2.patch against trunk revision 724229. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3686/console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I just committed this. Thanks, Boris!

          Show
          Tsz Wo Nicholas Sze added a comment - I just committed this. Thanks, Boris!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #683 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/683/)
          . Add NameNode storage information to the dfshealth page and move DataNode information to a separated page. (Boris Shkolnik via szetszwo)

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #683 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/683/ ) . Add NameNode storage information to the dfshealth page and move DataNode information to a separated page. (Boris Shkolnik via szetszwo)
          Hide
          Robert Chansler added a comment -

          Edit release note for publication.

          Show
          Robert Chansler added a comment - Edit release note for publication.

            People

            • Assignee:
              Boris Shkolnik
              Reporter:
              Robert Chansler
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development