HBase
  1. HBase
  2. HBASE-5912

org.apache.hadoop.hbase.ipc.ProtocolSignature.getFingerprint takes significant CPU

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 0.94.1
    • Fix Version/s: None
    • Component/s: IPC/RPC, Performance
    • Labels:
      None

      Description

      I ran oprofile on a YCSB client and found that a large percentage of the CPU time was going to this function:

      51991 0.4913 25361.jo java java.lang.reflect.Method[] java.lang.Class.copyMethods(java.lang.reflect.Method[])
      51384 0.4856 25361.jo java int org.apache.hadoop.hbase.ipc.ProtocolSignature.getFingerprint(java.lang.reflect.Method)
      50428 0.4766 25361.jo java void java.util.Arrays.sort1(int[], int, int)

      We should introduce a simple cache to avoid this overhead.

      1. hbase-5912.txt
        2 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        Adding a simple cache dropped the CPU time for my million-read workload on 0.94 from 1m40s to 1m30s, and improved average latency by a few % as well.

        Show
        Todd Lipcon added a comment - Adding a simple cache dropped the CPU time for my million-read workload on 0.94 from 1m40s to 1m30s, and improved average latency by a few % as well.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525251/hbase-5912.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
        org.apache.hadoop.hbase.master.TestSplitLogManager

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525251/hbase-5912.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1716//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        I think the test failures are unrelated, even though TestServerCustomProtocol sounds like something that would be related

        Show
        Todd Lipcon added a comment - I think the test failures are unrelated, even though TestServerCustomProtocol sounds like something that would be related
        Hide
        Jean-Daniel Cryans added a comment -

        Unmarking patch as available, it doesn't apply anymore since the code moved into sub modules.

        Show
        Jean-Daniel Cryans added a comment - Unmarking patch as available, it doesn't apply anymore since the code moved into sub modules.
        Hide
        stack added a comment -

        Resovling as no longer valid. We don't do protocol signature anymore (the unimplemented notion that if signatures didn't match, we'd volunteer some other protocol has all been stripped out). In fact all reflection has been undone from ipc.

        Show
        stack added a comment - Resovling as no longer valid. We don't do protocol signature anymore (the unimplemented notion that if signatures didn't match, we'd volunteer some other protocol has all been stripped out). In fact all reflection has been undone from ipc.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development