Hadoop Common
  1. Hadoop Common
  2. HADOOP-5258

Provide dfsadmin functionality to report on namenode's view of network topology

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New dfsAdmin command -printTopology shows topology as understood by the namenode.

      Description

      As discussed in HADOOP-4954, it would be useful to be able to query the namenode to its current view on the network topology of racks and datanodes. This would allow ops to compare what the namenode sees with what they expect it to see.

      1. HADOOP-5258.patch
        15 kB
        Jakob Homan
      2. HADOOP-5258.patch
        17 kB
        Jakob Homan
      3. HADOOP-5258.patch
        20 kB
        Jakob Homan
      4. HADOOP-5258.patch
        20 kB
        Jakob Homan

        Issue Links

          Activity

          Jakob Homan created issue -
          Jakob Homan made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-4954 [ HADOOP-4954 ]
          Hide
          Jakob Homan added a comment -

          If implemented, this feature would effectively allow 4954 to be performed.

          Show
          Jakob Homan added a comment - If implemented, this feature would effectively allow 4954 to be performed.
          Hide
          Jakob Homan added a comment -

          Patch:

          • Implements new dfsAdmin command -printTopology that prints out a tree of the racks and their attached nodes:
            Rack: /default-rack
               1.4.7.10:54726 (host1.domain.com)
               2.5.8.11:59142 (host2.domain.com)
               3.6.9.12:56643 (host3.domain.com)
               etc...
            

            This will allow the user to see the topology as the namenode currently sees it.

          • Provides a unit test for this new command. To test this I had to create a new type of test comparator in TestCLI, which is also included.
          • Moves a helper function that attempts to resolve an ip into a hostname from DatanoderReport to the utils class, as it was needed again for this patch.
          Show
          Jakob Homan added a comment - Patch: Implements new dfsAdmin command -printTopology that prints out a tree of the racks and their attached nodes: Rack: /default-rack 1.4.7.10:54726 (host1.domain.com) 2.5.8.11:59142 (host2.domain.com) 3.6.9.12:56643 (host3.domain.com) etc... This will allow the user to see the topology as the namenode currently sees it. Provides a unit test for this new command. To test this I had to create a new type of test comparator in TestCLI, which is also included. Moves a helper function that attempts to resolve an ip into a hostname from DatanoderReport to the utils class, as it was needed again for this patch.
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12400548 ]
          Hide
          Jakob Homan added a comment -

          Patch passes all unit tests in test and test-contrib except known-bad HADOOP-4907.

               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Jakob Homan added a comment - Patch passes all unit tests in test and test-contrib except known-bad HADOOP-4907 . [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Jakob Homan made changes -
          Release Note -printTopology added to dfsAdmin to print a tree of the rack and datanode topology as seen by the namenode.
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Jakob Homan added a comment -

          Canceling patch as forgot to update forrest documentation. Actual patch itself is fine, if someone wants to review.

          Show
          Jakob Homan added a comment - Canceling patch as forgot to update forrest documentation. Actual patch itself is fine, if someone wants to review.
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jakob Homan added a comment -

          Updated patch to update the forrest docs. Good to go.

          Show
          Jakob Homan added a comment - Updated patch to update the forrest docs. Good to go.
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12400558 ]
          Hide
          Jakob Homan added a comment -

          submitting updated patch

          Show
          Jakob Homan added a comment - submitting updated patch
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12400558/HADOOP-5258.patch
          against trunk revision 746166.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12400558/HADOOP-5258.patch against trunk revision 746166. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3887/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Sigh. All the tests that failed under Hudson pass on my machine. Two were obvious time outs, the other probably was as well.

          [junit] Running org.apache.hadoop.fs.TestCopyFiles
          [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 132.646 sec
          
          [junit] Running org.apache.hadoop.mapred.TestQueueManager
          [junit] Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 281.137 sec
          
          [junit] Running org.apache.hadoop.mapred.TestTaskLimits
          [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 89.81 sec
          
          Show
          Jakob Homan added a comment - Sigh. All the tests that failed under Hudson pass on my machine. Two were obvious time outs, the other probably was as well. [junit] Running org.apache.hadoop.fs.TestCopyFiles [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 132.646 sec [junit] Running org.apache.hadoop.mapred.TestQueueManager [junit] Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 281.137 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 89.81 sec
          Hide
          Konstantin Shvachko added a comment -
          1. StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil.
            See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method.
          2. In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here.
          3. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using
            Collections.binarySearch() and then List.add(int index, E element)
          4. In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case.
          5. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */
          Show
          Konstantin Shvachko added a comment - StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil . See org.apache.hadoop.net.NetUtils.normalizeHostName(String) , which is similar to your method. In DFSAdmin.printTopology() use ArrayList instead of Vector . Vector is synchronized, which is not necessary here. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element) In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */
          Hide
          Jakob Homan added a comment -

          canceling patch to address Konstantin's comments

          Show
          Jakob Homan added a comment - canceling patch to address Konstantin's comments
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Jakob Homan added a comment -

          StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil.

          Done. I had originally placed it in StringUtil because getHostName(), a similar method, was in StringUtil, but that method should also be in NetUtil. I've moved getHostName() to NetUtil, and updated the references to it.

          See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method.

          Actually, normalizeHostName(String) does the opposite of my method. It goes (hostname|IP) -> IP, mine goes IP -> hostname.

          In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here.

          Correct. The next item resolves this.

          Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element)

          I've replaced the ArrayList with a TreeSet, which resolves the issue and is more efficient than an ArrayList and cleaner than a binarySearch method.

          In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case.

          Done.

          DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */

          Done.

          Show
          Jakob Homan added a comment - StringUtils.getHostNameOfIP() should be a part of NetUtil rather than StringUtil. Done. I had originally placed it in StringUtil because getHostName(), a similar method, was in StringUtil, but that method should also be in NetUtil. I've moved getHostName() to NetUtil, and updated the references to it. See org.apache.hadoop.net.NetUtils.normalizeHostName(String), which is similar to your method. Actually, normalizeHostName(String) does the opposite of my method. It goes (hostname|IP) -> IP, mine goes IP -> hostname. In DFSAdmin.printTopology() use ArrayList instead of Vector. Vector is synchronized, which is not necessary here. Correct. The next item resolves this. Instead of explicitly sorting the data-nodes within the rack you could just insert them into the list in the right order, using Collections.binarySearch() and then List.add(int index, E element) I've replaced the ArrayList with a TreeSet, which resolves the issue and is more efficient than an ArrayList and cleaner than a binarySearch method. In DFSAdmin.printHelp() you should add printing printTopology in case cmd is not equal to any known command. In the very last else case. Done. DatanodeInfo.getNetworkLocation() could you please remove last * in the JavaDoc comment. Should be /** rack name */ Done.
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401265 ]
          Hide
          Jakob Homan added a comment -
               [exec] +1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 15 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          Patch passes all unit tests except the usual. Ready to go.

          Show
          Jakob Homan added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 15 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Patch passes all unit tests except the usual. Ready to go.
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12401265/HADOOP-5258.patch
          against trunk revision 749836.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401265/HADOOP-5258.patch against trunk revision 749836. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/41/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          +1

          Show
          Konstantin Shvachko added a comment - +1
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Jakob, there is javadoc warning (ant javadoc-dev) for the link to org.apache.hadoop.cli.TestCLI. Could you fix it?

          +   * Output tested by {@link org.apache.hadoop.cli.TestCLI}
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Jakob, there is javadoc warning (ant javadoc-dev) for the link to org.apache.hadoop.cli.TestCLI. Could you fix it? + * Output tested by {@link org.apache.hadoop.cli.TestCLI}
          Tsz Wo Nicholas Sze made changes -
          Component/s dfs [ 12310710 ]
          Hide
          Jakob Homan added a comment -

          Updating patch. Old one caused a Javadoc warning. That line has been removed.

          Show
          Jakob Homan added a comment - Updating patch. Old one caused a Javadoc warning. That line has been removed.
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401874 ]
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401874 ]
          Hide
          Jakob Homan added a comment -

          Let's try that again. File correctly updated.

          Show
          Jakob Homan added a comment - Let's try that again. File correctly updated.
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401875 ]
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401875 ]
          Jakob Homan made changes -
          Attachment HADOOP-5258.patch [ 12401876 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I committed this. Thanks, Jakob!

          Show
          Tsz Wo Nicholas Sze added a comment - I committed this. Thanks, Jakob!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )
          Jakob Homan made changes -
          Link This issue is related to HADOOP-5477 [ HADOOP-5477 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
          Robert Chansler made changes -
          Release Note -printTopology added to dfsAdmin to print a tree of the rack and datanode topology as seen by the namenode. New dfsAdmin command -printTopology shows topology as understood by the namenode.
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development