Hadoop Common
  1. Hadoop Common
  2. HADOOP-5258

Provide dfsadmin functionality to report on namenode's view of network topology

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New dfsAdmin command -printTopology shows topology as understood by the namenode.

      Description

      As discussed in HADOOP-4954, it would be useful to be able to query the namenode to its current view on the network topology of racks and datanodes. This would allow ops to compare what the namenode sees with what they expect it to see.

      1. HADOOP-5258.patch
        20 kB
        Jakob Homan
      2. HADOOP-5258.patch
        20 kB
        Jakob Homan
      3. HADOOP-5258.patch
        17 kB
        Jakob Homan
      4. HADOOP-5258.patch
        15 kB
        Jakob Homan

        Issue Links

          Activity

          Hide
          Jakob Homan added a comment -

          If implemented, this feature would effectively allow 4954 to be performed.

          Show
          Jakob Homan added a comment - If implemented, this feature would effectively allow 4954 to be performed.
          Hide
          Jakob Homan added a comment -

          Patch:

          • Implements new dfsAdmin command -printTopology that prints out a tree of the racks and their attached nodes:
            Rack: /default-rack
               1.4.7.10:54726 (host1.domain.com)
               2.5.8.11:59142 (host2.domain.com)
               3.6.9.12:56643 (host3.domain.com)
               etc...
            

            This will allow the user to see the topology as the namenode currently sees it.

          • Provides a unit test for this new command. To test this I had to create a new type of test comparator in TestCLI, which is also included.
          • Moves a helper function that attempts to resolve an ip into a hostname from DatanoderReport to the utils class, as it was needed again for this patch.
          Show
          Jakob Homan added a comment - Patch: Implements new dfsAdmin command -printTopology that prints out a tree of the racks and their attached nodes: Rack: /default-rack 1.4.7.10:54726 (host1.domain.com) 2.5.8.11:59142 (host2.domain.com) 3.6.9.12:56643 (host3.domain.com) etc... This will allow the user to see the topology as the namenode currently sees it. Provides a unit test for this new command. To test this I had to create a new type of test comparator in TestCLI, which is also included. Moves a helper function that attempts to resolve an ip into a hostname from DatanoderReport to the utils class, as it was needed again for this patch.
          Hide
          Jakob Homan added a comment -

          Patch passes all unit tests in test and test-contrib except known-bad HADOOP-4907.

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Jakob Homan added a comment - Patch passes all unit tests in test and test-contrib except known-bad HADOOP-4907 . [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Jakob Homan added a comment -

          Canceling patch as forgot to update forrest documentation. Actual patch itself is fine, if someone wants to review.

          Show
          Jakob Homan added a comment - Canceling patch as forgot to update forrest documentation. Actual patch itself is fine, if someone wants to review.
          Hide
          Jakob Homan added a comment -

          Updated patch to update the forrest docs. Good to go.

          Show
          Jakob Homan added a comment - Updated patch to update the forrest docs. Good to go.
          Hide
          Jakob Homan added a comment -

          submitting updated patch

          Show
          Jakob Homan added a comment - submitting updated patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12400558/HADOOP-5258.patch
          against trunk revision 746166.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400558/HADOOP-5258.patch against trunk revision 746166. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Sigh. All the tests that failed under Hudson pass on my machine. Two were obvious time outs, the other probably was as well.

          [junit] Running org.apache.hadoop.fs.TestCopyFiles
          [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 132.646 sec
          
          [junit] Running org.apache.hadoop.mapred.TestQueueManager
          [junit] Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 281.137 sec
          
          [junit] Running org.apache.hadoop.mapred.TestTaskLimits
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 89.81 sec
          
          Show
          Jakob Homan added a comment - Sigh. All the tests that failed under Hudson pass on my machine. Two were obvious time outs, the other probably was as well. [junit] Running org.apache.hadoop.fs.TestCopyFiles [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 132.646 sec [junit] Running org.apache.hadoop.mapred.TestQueueManager [junit] Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 281.137 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 89.81 sec
          Hide
          Konstantin Shvachko added a comment -
          1. StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil.
            See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method.
          2. In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here.
          3. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using
            Collections.binarySearch() and then List.add(int index, E element)
          4. In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case.
          5. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */
          Show
          Konstantin Shvachko added a comment - StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil . See org.apache.hadoop.net.NetUtils.normalizeHostName(String) , which is similar to your method. In DFSAdmin.printTopology() use ArrayList instead of Vector . Vector is synchronized, which is not necessary here. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element) In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */
          Hide
          Jakob Homan added a comment -

          canceling patch to address Konstantin's comments

          Show
          Jakob Homan added a comment - canceling patch to address Konstantin's comments
          Hide
          Jakob Homan added a comment -

          StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil.

          Done. I had originally placed it in StringUtil because getHostName(), a similar method, was in StringUtil, but that method should also be in NetUtil. I've moved getHostName() to NetUtil, and updated the references to it.

          See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method.

          Actually, normalizeHostName(String) does the opposite of my method. It goes (hostname|IP) -> IP, mine goes IP -> hostname.

          In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here.

          Correct. The next item resolves this.

          Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element)

          I've replaced the ArrayList with a TreeSet, which resolves the issue and is more efficient than an ArrayList and cleaner than a binarySearch method.

          In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case.

          Done.

          DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */

          Done.

          Show
          Jakob Homan added a comment - StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil. Done. I had originally placed it in StringUtil because getHostName(), a similar method, was in StringUtil, but that method should also be in NetUtil. I've moved getHostName() to NetUtil, and updated the references to it. See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method. Actually, normalizeHostName(String) does the opposite of my method. It goes (hostname|IP) -> IP, mine goes IP -> hostname. In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here. Correct. The next item resolves this. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element) I've replaced the ArrayList with a TreeSet, which resolves the issue and is more efficient than an ArrayList and cleaner than a binarySearch method. In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case. Done. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */ Done.
          Hide
          Jakob Homan added a comment -
               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Patch passes all unit tests except the usual. Ready to go.

          Show
          Jakob Homan added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Patch passes all unit tests except the usual. Ready to go.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12401265/HADOOP-5258.patch
          against trunk revision 749836.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401265/HADOOP-5258.patch against trunk revision 749836. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          +1

          Show
          Konstantin Shvachko added a comment - +1
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Jakob, there is javadoc warning (ant javadoc-dev) for the link to org.apache.hadoop.cli.TestCLI. Could you fix it?

          +   * Output tested by {@link org.apache.hadoop.cli.TestCLI}
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Jakob, there is javadoc warning (ant javadoc-dev) for the link to org.apache.hadoop.cli.TestCLI. Could you fix it? + * Output tested by {@link org.apache.hadoop.cli.TestCLI}
          Hide
          Jakob Homan added a comment -

          Updating patch. Old one caused a Javadoc warning. That line has been removed.

          Show
          Jakob Homan added a comment - Updating patch. Old one caused a Javadoc warning. That line has been removed.
          Hide
          Jakob Homan added a comment -

          Let's try that again. File correctly updated.

          Show
          Jakob Homan added a comment - Let's try that again. File correctly updated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I committed this. Thanks, Jakob!

          Show
          Tsz Wo Nicholas Sze added a comment - I committed this. Thanks, Jakob!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development