Hadoop Common
  1. Hadoop Common
  2. HADOOP-4756

Create a command line tool to access JMX exported properties from a NameNode server

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      New HDFS tool JMXGet facilitates command line access to statistics via JMX.

      Description

      Create a command line tool that will easy script access to JMX exported properties of the NameNode.

      1. HADOOP-4756-4.patch
        17 kB
        Boris Shkolnik
      2. HADOOP-4756-3.patch
        17 kB
        Boris Shkolnik
      3. HADOOP-4756-2.patch
        16 kB
        Boris Shkolnik
      4. HADOOP-4756-1.patch
        17 kB
        Boris Shkolnik
      5. HADOOP-4756.patch
        11 kB
        Boris Shkolnik
      6. HADOOP-4756.patch
        13 kB
        Boris Shkolnik
      7. HADOOP-4756.patch
        12 kB
        Boris Shkolnik

        Activity

        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
        Hide
        Raghu Angadi added a comment -

        Trunk (0.21). Just updated "Fix Version".

        Show
        Raghu Angadi added a comment - Trunk (0.21). Just updated "Fix Version".
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Raghu, which versions is this committed to?

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Raghu, which versions is this committed to?
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/)
        . A command line tool to access JMX properties on NameNode
        and DataNode. (Boris Shkolnik via rangadi)

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ ) . A command line tool to access JMX properties on NameNode and DataNode. (Boris Shkolnik via rangadi)
        Hide
        Raghu Angadi added a comment -

        I just committed this, Thanks Boris!

        (Release note could be changed to inform about the new tool).

        Show
        Raghu Angadi added a comment - I just committed this, Thanks Boris! (Release note could be changed to inform about the new tool).
        Hide
        Boris Shkolnik added a comment -

        fixed the lines lengths

        Show
        Boris Shkolnik added a comment - fixed the lines lengths
        Hide
        Raghu Angadi added a comment -

        I just mentioned that as part of some related discussion.. not a -1 from me... but I sure prefer lines to fit well within 80 columns.

        Btw, this jira is a very good example of detailed review and equally importantly detailed reply to the review.

        Show
        Raghu Angadi added a comment - I just mentioned that as part of some related discussion.. not a -1 from me... but I sure prefer lines to fit well within 80 columns. Btw, this jira is a very good example of detailed review and equally importantly detailed reply to the review.
        Hide
        Suresh Srinivas added a comment -

        Raghu pointed out that the patch includes some lines that are longer than 80 chars. Other than that +1 for the patch.

        Show
        Suresh Srinivas added a comment - Raghu pointed out that the patch includes some lines that are longer than 80 chars. Other than that +1 for the patch.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12401786/HADOOP-4756-3.patch
        against trunk revision 752073.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401786/HADOOP-4756-3.patch against trunk revision 752073. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/65/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment - - edited

        Some minor nits:

        1. JMXGet.getValue() err() print statement can be simply changed to err("Info: key = " + key + "; val = " + val);

          Done

        2. JMXGet.init() always returns zero. So it could be made to return void

          Done

        3. JMXGet.parseArgs() it may be a good idea to use static methods invoked with class name so that code looks clean on eclipse. Typo invalide in exception. Not sure if not catching the IllegalArgumentException from parseArgs() in main() is intentional. Does this return -1 from the script?

          Done

        4. JMXGet.main() null value is not printed. Could null be a valid value that needs to be printed?

          No. Null is for the case it is invalid key

        5. TestJMXGet.java does not compile because of the last line. Also it might be a good idea to include tests for running the command with server:port and localVM option.

          Fixed.

        Warning from FindBug is fixed too.

        Show
        Boris Shkolnik added a comment - - edited Some minor nits: JMXGet.getValue() err() print statement can be simply changed to err("Info: key = " + key + "; val = " + val); Done JMXGet.init() always returns zero. So it could be made to return void Done JMXGet.parseArgs() it may be a good idea to use static methods invoked with class name so that code looks clean on eclipse. Typo invalide in exception. Not sure if not catching the IllegalArgumentException from parseArgs() in main() is intentional. Does this return -1 from the script? Done JMXGet.main() null value is not printed. Could null be a valid value that needs to be printed? No. Null is for the case it is invalid key TestJMXGet.java does not compile because of the last line. Also it might be a good idea to include tests for running the command with server:port and localVM option. Fixed. Warning from FindBug is fixed too.
        Hide
        Suresh Srinivas added a comment -

        Some minor nits:

        1. JMXGet.getValue() err() print statement can be simply changed to err("Info: key = " + key + "; val = " + val);
        2. JMXGet.init() always returns zero. So it could be made to return void
        3. JMXGet.parseArgs() it may be a good idea to use static methods invoked with class name so that code looks clean on eclipse. Typo invalide in exception. Not sure if not catching the IllegalArgumentException from parseArgs() in main() is intentional. Does this return -1 from the script?
        4. JMXGet.main() null value is not printed. Could null be a valid value that needs to be printed?
        5. TestJMXGet.java does not compile because of the last line. Also it might be a good idea to include tests for running the command with server:port and localVM option.
        Show
        Suresh Srinivas added a comment - Some minor nits: JMXGet.getValue() err() print statement can be simply changed to err("Info: key = " + key + "; val = " + val); JMXGet.init() always returns zero. So it could be made to return void JMXGet.parseArgs() it may be a good idea to use static methods invoked with class name so that code looks clean on eclipse. Typo invalide in exception. Not sure if not catching the IllegalArgumentException from parseArgs() in main() is intentional. Does this return -1 from the script? JMXGet.main() null value is not printed. Could null be a valid value that needs to be printed? TestJMXGet.java does not compile because of the last line. Also it might be a good idea to include tests for running the command with server:port and localVM option.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12401560/HADOOP-4756-2.patch
        against trunk revision 751463.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401560/HADOOP-4756-2.patch against trunk revision 751463. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/29/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment -
        1. General
          1. nits - Coding conventions - have consistent convention for space before and after assignemtn =). Space is required after if, for, , and parenthesis around the if statement.

            Done

          2. debug logs are sent to err. Not sure if that is the right thing to do.

            This is a script. It should write either to stdout or stderr. I want user to be able to separate logs from the output.

          3. Where possible, instead of catching Exception, catch more specific exceptions expected

            Changed to specific Exceptions where possible. But some of JMX functions can throw many different not related exceptions so I need to use Exception.

          4. There is a difference in exceptions caught in getValue compared printAllValues(). Should they be the same?

            Fixed

        2. hdfs
          1. Add new command jmxget to the usage

            Done

        3. JMXGet.java
          1. Following imports are not used:
            import org.apache.hadoop.conf.Configuration;
            import org.apache.hadoop.conf.Configured;
            import org.apache.hadoop.util.Tool;
            import org.apache.hadoop.util.ToolRunner;
            

            Done

          2. Move member variables to the top of the class

            Done

          3. main() Like other commands such as DFSAdmin.java should we use ToolRunner to start this tool?

            No. I tried to make the script independent of hadoop code. So it can be run on any machine.

          4. JMXGet() Constructor comments are not right - there is no conf param.

            Fixed

          5. run() Method comments are not right. There is not parameter args

            Fixed

          6. parseArgs() instead of catching ParserException, should it be thrown? That way init() or run() can catch it and do the required error processing

            I rewritten this piece.

          7. init()
            1. Should only throw IOException not generic Exception

              See comment about exceptions above

            2. defined port, server as static String variables such as DEFAULT_PORT, DEFAULT_SERVER

              Removed

            3. No need to catch exception while doing JMXConnectorFactory.connect(), as it can be caught by run()

              Changed

            4. Remove commented code in the for loop

              Done

          8. printUsage() method comment says GMX instead of JMX

            Done

          9. printAllValues() should method comment be saying Print all the attribute values?

            Done

        Show
        Boris Shkolnik added a comment - General nits - Coding conventions - have consistent convention for space before and after assignemtn = ). Space is required after if , for , , and parenthesis around the if statement. Done debug logs are sent to err . Not sure if that is the right thing to do. This is a script. It should write either to stdout or stderr. I want user to be able to separate logs from the output. Where possible, instead of catching Exception , catch more specific exceptions expected Changed to specific Exceptions where possible. But some of JMX functions can throw many different not related exceptions so I need to use Exception. There is a difference in exceptions caught in getValue compared printAllValues() . Should they be the same? Fixed hdfs Add new command jmxget to the usage Done JMXGet.java Following imports are not used: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; Done Move member variables to the top of the class Done main() Like other commands such as DFSAdmin.java should we use ToolRunner to start this tool? No. I tried to make the script independent of hadoop code. So it can be run on any machine. JMXGet() Constructor comments are not right - there is no conf param. Fixed run() Method comments are not right. There is not parameter args Fixed parseArgs() instead of catching ParserException, should it be thrown? That way init() or run() can catch it and do the required error processing I rewritten this piece. init() Should only throw IOException not generic Exception See comment about exceptions above defined port , server as static String variables such as DEFAULT_PORT , DEFAULT_SERVER Removed No need to catch exception while doing JMXConnectorFactory.connect() , as it can be caught by run() Changed Remove commented code in the for loop Done printUsage() method comment says GMX instead of JMX Done printAllValues() should method comment be saying Print all the attribute values ? Done
        Hide
        Boris Shkolnik added a comment -

        Regarding Hudson comments:
        1. Failed test is unrelated (timeout in some chukwa test).
        2. 2 findbugs. Fixed.

        Show
        Boris Shkolnik added a comment - Regarding Hudson comments: 1. Failed test is unrelated (timeout in some chukwa test). 2. 2 findbugs. Fixed.
        Hide
        Eric Yang added a comment -

        Resubmit patch to hudson, trunk test was broken by HADOOP-5409.

        Show
        Eric Yang added a comment - Resubmit patch to hudson, trunk test was broken by HADOOP-5409 .
        Hide
        Boris Shkolnik added a comment -

        You can definitely connect using Jconsole (works fine) for remote monitoring, but the purpose of this script is to allow writing other scripts which need to get data from the Name/Data Node.

        Show
        Boris Shkolnik added a comment - You can definitely connect using Jconsole (works fine) for remote monitoring, but the purpose of this script is to allow writing other scripts which need to get data from the Name/Data Node.
        Hide
        He Yongqiang added a comment -

        Can we use jconsole or other tools for this?

        I remember that in hadoop 0.17, jconsole can not connect to hadoop NameNode daemon, DataNode daemon, because there is no JMXConnectorServer in there. So in hadoop 0.17, what I did is to add some JMXConectorServer code myslef, with which i can use jconsol for monitoring. Because those code is not licensice compatible, so i did not commit that.
        Has that been added officially now?

        Show
        He Yongqiang added a comment - Can we use jconsole or other tools for this? I remember that in hadoop 0.17, jconsole can not connect to hadoop NameNode daemon, DataNode daemon, because there is no JMXConnectorServer in there. So in hadoop 0.17, what I did is to add some JMXConectorServer code myslef, with which i can use jconsol for monitoring. Because those code is not licensice compatible, so i did not commit that. Has that been added officially now?
        Hide
        Suresh Srinivas added a comment -
        1. General
          1. nits - Coding conventions - have consistent convention for space before and after assignemtn =). Space is required after if, for, , and parenthesis around the if statement.
          2. debug logs are sent to err. Not sure if that is the right thing to do.
          3. Where possible, instead of catching Exception, catch more specific exceptions expected
          4. There is a difference in exceptions caught in getValue compared printAllValues(). Should they be the same?
        2. hdfs
          1. Add new command jmxget to the usage
        3. JMXGet.java
          1. Following imports are not used:
            import org.apache.hadoop.conf.Configuration;
            import org.apache.hadoop.conf.Configured;
            import org.apache.hadoop.util.Tool;
            import org.apache.hadoop.util.ToolRunner;
            
          2. Move member variables to the top of the class
          3. main() Like other commands such as DFSAdmin.java should we use ToolRunner to start this tool?
          4. JMXGet() Constructor comments are not right - there is no conf param.
          5. run() Method comments are not right. There is not parameter args
          6. parseArgs() instead of catching ParserException, should it be thrown? That way init() or run() can catch it and do the required error processing
          7. init()
            1. Should only throw IOException not generic Exception
            2. defined port, server as static String variables such as DEFAULT_PORT, DEFAULT_SERVER
            3. No need to catch exception while doing JMXConnectorFactory.connect(), as it can be caught by run()
            4. Remove commented code in the for loop
          8. printUsage() method comment says GMX instead of JMX
          9. printAllValues() should method comment be saying Print all the attribute values?
        Show
        Suresh Srinivas added a comment - General nits - Coding conventions - have consistent convention for space before and after assignemtn = ). Space is required after if , for , , and parenthesis around the if statement. debug logs are sent to err . Not sure if that is the right thing to do. Where possible, instead of catching Exception , catch more specific exceptions expected There is a difference in exceptions caught in getValue compared printAllValues() . Should they be the same? hdfs Add new command jmxget to the usage JMXGet.java Following imports are not used: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; Move member variables to the top of the class main() Like other commands such as DFSAdmin.java should we use ToolRunner to start this tool? JMXGet() Constructor comments are not right - there is no conf param. run() Method comments are not right. There is not parameter args parseArgs() instead of catching ParserException, should it be thrown? That way init() or run() can catch it and do the required error processing init() Should only throw IOException not generic Exception defined port , server as static String variables such as DEFAULT_PORT , DEFAULT_SERVER No need to catch exception while doing JMXConnectorFactory.connect() , as it can be caught by run() Remove commented code in the for loop printUsage() method comment says GMX instead of JMX printAllValues() should method comment be saying Print all the attribute values ?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12401155/HADOOP-4756.patch
        against trunk revision 748861.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12401155/HADOOP-4756.patch against trunk revision 748861. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/30/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment - - edited

        modified to use DynamicMBean aproach, for all the MBeans. it makes the implementation much more straight forward. And it should work for both Name and Data Nodes.

        Show
        Boris Shkolnik added a comment - - edited modified to use DynamicMBean aproach, for all the MBeans. it makes the implementation much more straight forward. And it should work for both Name and Data Nodes.

          People

          • Assignee:
            Boris Shkolnik
            Reporter:
            Boris Shkolnik
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development