HBase
  1. HBase
  2. HBASE-6602

Region Server Dynamic Metrics can cause high cpu usage.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.

      Seeing jstacks like this:

      "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
      java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.put(HashMap.java:374)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)

      at java.util.TimerThread.run(Timer.java:462)

      "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
      java.lang.Thread.State: RUNNABLE
      at java.util.HashMap.put(HashMap.java:374)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
      at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
      at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)

      1. HBASE-6602-0.94-0.patch
        3 kB
        Elliott Clark
      2. HBASE-6602-trunk-0.patch
        3 kB
        stack

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/)
        HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/ ) HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/)
        HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #48 (See https://builds.apache.org/job/HBase-0.94-security/48/ ) HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #406 (See https://builds.apache.org/job/HBase-0.94/406/)
        HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #406 (See https://builds.apache.org/job/HBase-0.94/406/ ) HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3233 (See https://builds.apache.org/job/HBase-TRUNK/3233/)
        HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374397)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3233 (See https://builds.apache.org/job/HBase-TRUNK/3233/ ) HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374397) Result = SUCCESS stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
        Hide
        stack added a comment -

        Committed to 0.94 and trunk. Thanks for the patch Mr. Elliott.

        Show
        stack added a comment - Committed to 0.94 and trunk. Thanks for the patch Mr. Elliott.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541386/HBASE-6602-trunk-0.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541386/HBASE-6602-trunk-0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//console This message is automatically generated.
        Hide
        stack added a comment -

        Trunk version.

        Show
        stack added a comment - Trunk version.
        Hide
        Ted Yu added a comment -

        I am fine with the current naming.

        Show
        Ted Yu added a comment - I am fine with the current naming.
        Hide
        stack added a comment -

        0.94.2 and critical because can end up burning loads of CPU

        Show
        stack added a comment - 0.94.2 and critical because can end up burning loads of CPU
        Hide
        stack added a comment -

        +1

        I asked Elliott if flag needs to be a data member rather than local variable and he thought about having methods return true/false if needs update... but turns out the interaction w/ clear requires that it be a data member.

        needsUpdateMessage seems fine to me.

        Show
        stack added a comment - +1 I asked Elliott if flag needs to be a data member rather than local variable and he thought about having methods return true/false if needs update... but turns out the interaction w/ clear requires that it be a data member. needsUpdateMessage seems fine to me.
        Hide
        Elliott Clark added a comment -

        The term message implies the async nature of function call so I prefer needsUpdateMessage; I can change it if you're stuck on that.

        Show
        Elliott Clark added a comment - The term message implies the async nature of function call so I prefer needsUpdateMessage; I can change it if you're stuck on that.
        Hide
        Ted Yu added a comment -

        Patch looks good.
        Maybe renaming needsUpdateMessage as metricsNeedUpdate.

        Show
        Ted Yu added a comment - Patch looks good. Maybe renaming needsUpdateMessage as metricsNeedUpdate.
        Hide
        Elliott Clark added a comment -

        Attaching the 0.94 branch patch as hopefully we will move fully over to metrics2 and this issue will go away.

        Show
        Elliott Clark added a comment - Attaching the 0.94 branch patch as hopefully we will move fully over to metrics2 and this issue will go away.

          People

          • Assignee:
            Elliott Clark
            Reporter:
            Elliott Clark
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development