HBase
  1. HBase
  2. HBASE-1722

Add support for exporting HBase metrics via JMX

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.19.3
    • Fix Version/s: 0.20.1
    • Component/s: IPC/RPC, master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Current HBase metrics collection and reporting (for the 0.19 releases at least) doesn't seem to export any of the metrics as MBeans via JMX.

      This patch adds JMX MBean support for reporting metrics collected in 3 areas:

      • org.apache.hadoop.hbase.ipc.HBaseRpcMetrics
      • org.apache.hadoop.hbase.master.metrics.MasterMetrics
      • org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics

      This patch is only against the 0.19 branch, as the metrics API seems to have changed a bit in Hadoop 0.20, including adding support for dynamic MBeans. I'll add a patch reworked for trunk shortly.

      1. metrics_docs_trunk.patch
        4 kB
        Gary Helmling
      2. metrics_docs_0.19.patch
        4 kB
        Gary Helmling
      3. jmx.patch
        9 kB
        stack
      4. jmx_support-trunk.patch
        10 kB
        Gary Helmling
      5. jmx_metrics-0.19.patch
        19 kB
        Gary Helmling
      6. hbase-bin.patch
        4 kB
        Gary Helmling

        Activity

        Hide
        Gary Helmling added a comment -

        This patch provide JMX support against the current 0.19 branch, adding a JMX MBean interface and implementing class for each of the 3 metrics areas in the issue.

        Please let me know of any problems, code style issues, etc.

        Show
        Gary Helmling added a comment - This patch provide JMX support against the current 0.19 branch, adding a JMX MBean interface and implementing class for each of the 3 metrics areas in the issue. Please let me know of any problems, code style issues, etc.
        Hide
        stack added a comment -

        Thanks for the nice patch Gary. Any chance of one for hbase TRUNK with maybe a bit of documentation on how to set things up so metrics show in something like jconsole? Only issue was missing Apache copyright. In future, just watch line lengths. Hadoop project is 80 characters generally.

        Show
        stack added a comment - Thanks for the nice patch Gary. Any chance of one for hbase TRUNK with maybe a bit of documentation on how to set things up so metrics show in something like jconsole? Only issue was missing Apache copyright. In future, just watch line lengths. Hadoop project is 80 characters generally.
        Hide
        Gary Helmling added a comment -

        Here's a patch against current trunk. This one was generated with git, but I was able to test apply cleanly with patch -p1.

        This one was even easier thanks to MetricsDynamicMBeanBase in hadoop 0.20! I was able to drop the need for separate interfaces and pass through accessors for each metric.

        I'll put up some docs on enable JMX monitoring on the wiki. For remote monitoring this may require some tweaks to hbase-env.sh and the hbase shell script – like separating out option vars for master vs. regionservers.

        Show
        Gary Helmling added a comment - Here's a patch against current trunk. This one was generated with git, but I was able to test apply cleanly with patch -p1. This one was even easier thanks to MetricsDynamicMBeanBase in hadoop 0.20! I was able to drop the need for separate interfaces and pass through accessors for each metric. I'll put up some docs on enable JMX monitoring on the wiki. For remote monitoring this may require some tweaks to hbase-env.sh and the hbase shell script – like separating out option vars for master vs. regionservers.
        Hide
        stack added a comment -

        Patch looks good.

        On docs, add your comment here, http://hadoop.apache.org/hbase/docs/r0.19.3/metrics.html, rather than to wiki. If its checked in as part of source, doc gets packaged with the version it targets. To find the page to edit, its at src/docs/src/documentation/content/xdocs/metrics.xml in the repository.

        Thanks Gary.

        Show
        stack added a comment - Patch looks good. On docs, add your comment here, http://hadoop.apache.org/hbase/docs/r0.19.3/metrics.html , rather than to wiki. If its checked in as part of source, doc gets packaged with the version it targets. To find the page to edit, its at src/docs/src/documentation/content/xdocs/metrics.xml in the repository. Thanks Gary.
        Hide
        stack added a comment -

        Reopen till we do commit against TRUNK

        Show
        stack added a comment - Reopen till we do commit against TRUNK
        Hide
        Gary Helmling added a comment -

        Ok, I'll do a write up in metrics.xml instead and post patches for that against 0.19 and trunk. Thanks for the quick review!

        Show
        Gary Helmling added a comment - Ok, I'll do a write up in metrics.xml instead and post patches for that against 0.19 and trunk. Thanks for the quick review!
        Hide
        Gary Helmling added a comment -

        Here's a patch for metrics.xml in trunk.

        This is my first time using Forrest, so let me know if anything seems off or missing.

        The last bit about editing bin/hbase is kind of hacky, but is necessary since we currently don't split out opts for master vs. regionserver vs. shell, etc.

        I'll attach a separate version for 0.19 – the only real difference is the domain where the MBeans show up.

        Show
        Gary Helmling added a comment - Here's a patch for metrics.xml in trunk. This is my first time using Forrest, so let me know if anything seems off or missing. The last bit about editing bin/hbase is kind of hacky, but is necessary since we currently don't split out opts for master vs. regionserver vs. shell, etc. I'll attach a separate version for 0.19 – the only real difference is the domain where the MBeans show up.
        Hide
        Gary Helmling added a comment -

        Here is a patch for metrics.xml against 0.19. The only change is the domain for where the MBeans appear (hadoop.dfs) – this is controlled by the Hadoop MBeanUtil class.

        Again if anything's missing or additional information would be useful, let me know.

        Show
        Gary Helmling added a comment - Here is a patch for metrics.xml against 0.19. The only change is the domain for where the MBeans appear (hadoop.dfs) – this is controlled by the Hadoop MBeanUtil class. Again if anything's missing or additional information would be useful, let me know.
        Hide
        stack added a comment -

        @Gary:

        I reviewed patch for TRUNK. Its great. I like the addition to monitoring docs particularly. Before applying, do you want to add in JMX environment variables to hbase-env.sh and include them in bin/hbase.sh rather than have users copy and paste out of the monitoring documentation? (Default, the JMX env variables can be null).

        Otherwise, was going to apply to TRUNK and branch so would be in 0.20.1.

        Thanks.

        Show
        stack added a comment - @Gary: I reviewed patch for TRUNK. Its great. I like the addition to monitoring docs particularly. Before applying, do you want to add in JMX environment variables to hbase-env.sh and include them in bin/hbase.sh rather than have users copy and paste out of the monitoring documentation? (Default, the JMX env variables can be null). Otherwise, was going to apply to TRUNK and branch so would be in 0.20.1. Thanks.
        Hide
        Gary Helmling added a comment -

        Here's a patch for the bin/hbase and conf/hbase-env.sh scripts and conf/hadoop-metrics.properties, adding placeholder configurations to ease the process of enabling JMX monitoring. If the conditional checking in bin/hbase for a "start" action is too weird though, feel free to skip these changes. The config is documented in the metrics.xml patch. This just provides a bit more guidance for enabling.

        Show
        Gary Helmling added a comment - Here's a patch for the bin/hbase and conf/hbase-env.sh scripts and conf/hadoop-metrics.properties, adding placeholder configurations to ease the process of enabling JMX monitoring. If the conditional checking in bin/hbase for a "start" action is too weird though, feel free to skip these changes. The config is documented in the metrics.xml patch. This just provides a bit more guidance for enabling.
        Hide
        stack added a comment -

        Thanks for the new patch Gary. I took a look at it. I like the addition of HBASE_REGIONSERVER_OPTIONS, etc., only I think itneeds more work; it works for setting jmx options per daemon but would be sweet if could set jvm mx and ms per daemon (and for zookeeper) as per HBASE-1687. Its a little tricky because we have that HBASE_HEAPSIZE thingy.

        But no biggie. You have done the work documenting how to make jmx run with your additions to the metrics page. Let me commit. Folks who want to jmx can just read your metrics additions. 'll commit to TRUNK. I won't add to branch, unless you insist. Lets fix the above in a new issue or as part of HBASE-1687.

        When we add suggested jmx options to hbase-env.sh, maybe default should be no default? Its a bit easier than pointing to a file with user login and password for jconsole?

        Good stuff Gary.

        Show
        stack added a comment - Thanks for the new patch Gary. I took a look at it. I like the addition of HBASE_REGIONSERVER_OPTIONS, etc., only I think itneeds more work; it works for setting jmx options per daemon but would be sweet if could set jvm mx and ms per daemon (and for zookeeper) as per HBASE-1687 . Its a little tricky because we have that HBASE_HEAPSIZE thingy. But no biggie. You have done the work documenting how to make jmx run with your additions to the metrics page. Let me commit. Folks who want to jmx can just read your metrics additions. 'll commit to TRUNK. I won't add to branch, unless you insist. Lets fix the above in a new issue or as part of HBASE-1687 . When we add suggested jmx options to hbase-env.sh, maybe default should be no default? Its a bit easier than pointing to a file with user login and password for jconsole? Good stuff Gary.
        Hide
        stack added a comment -

        Roll up of Gary's classes – Its what I committed to TRUNK.

        Show
        stack added a comment - Roll up of Gary's classes – Its what I committed to TRUNK.
        Hide
        stack added a comment -

        Moving to TRUNK

        Show
        stack added a comment - Moving to TRUNK
        Hide
        stack added a comment -

        Update: I committed patch added by Ryan to hbase-1687. His patch was almost exactly the content of hbase-bin.patch above. My reasons for omitting hbase-bin.patch didn't hold up under scrutiny. The HBASE_HEAPSIZE appears before the HBASE_*_OPTS added by his patch (and by hbase-bin.patch) and because the last -Xmx/-Xms wins, adding -Xmx/-Xms that override HBASE_HEAPSIZE works.

        On commit of HBASE-1687, I amended the metrics.xml documentation to remove mention of HBASE_*_OPTS addition now its no longer needed.

        Show
        stack added a comment - Update: I committed patch added by Ryan to hbase-1687. His patch was almost exactly the content of hbase-bin.patch above. My reasons for omitting hbase-bin.patch didn't hold up under scrutiny. The HBASE_HEAPSIZE appears before the HBASE_*_OPTS added by his patch (and by hbase-bin.patch) and because the last -Xmx/-Xms wins, adding -Xmx/-Xms that override HBASE_HEAPSIZE works. On commit of HBASE-1687 , I amended the metrics.xml documentation to remove mention of HBASE_*_OPTS addition now its no longer needed.
        Hide
        Gary Helmling added a comment -

        Excellent! Thanks, stack!

        Show
        Gary Helmling added a comment - Excellent! Thanks, stack!
        Hide
        stack added a comment -

        Committed to TRUNK. Thanks for the nice patch Gary.

        Show
        stack added a comment - Committed to TRUNK. Thanks for the nice patch Gary.
        Hide
        stack added a comment -

        Backport to 0.20 branch.

        Show
        stack added a comment - Backport to 0.20 branch.
        Hide
        stack added a comment -

        I backported this patch and the metrics jmx doc. to 0.20.1.

        Show
        stack added a comment - I backported this patch and the metrics jmx doc. to 0.20.1.

          People

          • Assignee:
            Gary Helmling
            Reporter:
            Gary Helmling
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development