Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23185

High cpu usage because getTable()#put() gets config value every time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0, 1.4.10, 1.2.12, 1.3.5
    • Fix Version/s: 1.6.0, 1.4.12, 1.3.7, 1.6.1
    • Component/s: Client
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      When we analyzed the performance of our hbase application with many puts, we found that Configuration methods use many CPU resources:

      As you can see, getTable().put() is calling Configuration methods which cause regex or synchronization by Hashtable.

      This should not happen in 0.99.2 because https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue.
      However, it's reproducing nowadays by bugs or leakages after many code evoluations between 0.9x and 1.x.

      1. https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374
        • finishSetup is called every new HTable() e.g. every con.getTable()
        • So getInt is called everytime and it does regex
      2. https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115
        • BufferedMutatorImpl is created every first put for HTable e.g. con.getTable().put()
        • Create ConnectionConf every time in BufferedMutatorImpl constructor
        • ConnectionConf gets config value in the constructor
      3. https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326
        • AsyncProcess is created in BufferedMutatorImpl constructor, so new AsyncProcess is created by con.getTable().put()
        • AsyncProcess parse many configurations

      So, con.getTable().put() is heavy operation for CPU because of getting config value.

       

      With in-house patch for this issue, we observed about 10% improvement on max-throughput (e.g. CPU usage) at client-side:

       

      Seems branch-2 is not affected because client implementation has been changed dramatically.
       

        Attachments

        1. Screenshot from 2019-10-18 12-38-14.png
          270 kB
          Shinya Yoshida
        2. Screenshot from 2019-10-18 13-03-24.png
          65 kB
          Shinya Yoshida

          Activity

            People

            • Assignee:
              lineyshinya Shinya Yoshida
              Reporter:
              lineyshinya Shinya Yoshida
            • Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: