Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6602

Region Server Dynamic Metrics can cause high cpu usage.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.

      Seeing jstacks like this:

      "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
      java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.put(HashMap.java:374)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)

      at java.util.TimerThread.run(Timer.java:462)

      "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
      java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.put(HashMap.java:374)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
      at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)

        Attachments

        1. HBASE-6602-0.94-0.patch
          3 kB
          Elliott Clark
        2. HBASE-6602-trunk-0.patch
          3 kB
          stack

          Activity

            People

            • Assignee:
              eclark Elliott Clark
              Reporter:
              eclark Elliott Clark
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: