HBase
  1. HBase
  2. HBASE-5901

Use union type protobufs instead of class/byte pairs for multi requests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: IPC/RPC, Performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The current implementation of multi actions uses repeated "NameBytesPair"s for the contents of multi actions. Instead, we should introduce a union type protobuf for the valid actions. This makes the RPCs smaller since they don't need to carry class names, and makes deserialization faster since it can avoid some copying and reflection.

      1. hbase-5901.txt
        58 kB
        stack
      2. hbase-5901.txt
        58 kB
        Todd Lipcon
      3. hbase-5901.txt
        59 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        This patch dropped cumulative CPU usage by about 10% for a million-record insert.

        Show
        Todd Lipcon added a comment - This patch dropped cumulative CPU usage by about 10% for a million-record insert.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525044/hbase-5901.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.util.TestHBaseFsck

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525044/hbase-5901.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.util.TestHBaseFsck Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1687//console This message is automatically generated.
        Hide
        stack added a comment -

        +1

        Nice.

        Show
        stack added a comment - +1 Nice.
        Hide
        Jimmy Xiang added a comment -

        +1

        Show
        Jimmy Xiang added a comment - +1
        Hide
        Todd Lipcon added a comment -

        Patch rebased against trunk (something else conflicted with it)

        Show
        Todd Lipcon added a comment - Patch rebased against trunk (something else conflicted with it)
        Hide
        stack added a comment -

        @Todd you going to commit? I can otherwise.

        Show
        stack added a comment - @Todd you going to commit? I can otherwise.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525223/hbase-5901.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.client.TestFromClientSide
        org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525223/hbase-5901.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1712//console This message is automatically generated.
        Hide
        stack added a comment -

        Retrying. I tried the failing tests local and they pass for me.

        Show
        stack added a comment - Retrying. I tried the failing tests local and they pass for me.
        Hide
        Todd Lipcon added a comment -

        I tried the same locally. I'll commit this since it seems the test failures are something else.

        Show
        Todd Lipcon added a comment - I tried the same locally. I'll commit this since it seems the test failures are something else.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2835 (See https://builds.apache.org/job/HBase-TRUNK/2835/)
        HBASE-5901. Use union type protobufs instead of class/byte pairs for multi requests. (Revision 1332882)

        Result = SUCCESS
        todd :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java
        • /hbase/trunk/src/main/protobuf/Client.proto
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2835 (See https://builds.apache.org/job/HBase-TRUNK/2835/ ) HBASE-5901 . Use union type protobufs instead of class/byte pairs for multi requests. (Revision 1332882) Result = SUCCESS todd : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java /hbase/trunk/src/main/protobuf/Client.proto
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #189 (See https://builds.apache.org/job/HBase-TRUNK-security/189/)
        HBASE-5901. Use union type protobufs instead of class/byte pairs for multi requests. (Revision 1332882)

        Result = SUCCESS
        todd :
        Files :

        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java
        • /hbase/trunk/src/main/protobuf/Client.proto
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #189 (See https://builds.apache.org/job/HBase-TRUNK-security/189/ ) HBASE-5901 . Use union type protobufs instead of class/byte pairs for multi requests. (Revision 1332882) Result = SUCCESS todd : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java /hbase/trunk/src/main/protobuf/Client.proto
        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development