HBase
  1. HBase
  2. HBASE-2182

rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.

    Details

    • Type: Umbrella Umbrella
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: IPC/RPC, Performance
    • Labels:
      None

      Description

      Reimplement HBase RPC with an NIO framework.

        Issue Links

          Activity

          Andrew Purtell created issue -
          Andrew Purtell made changes -
          Field Original Value New Value
          Link This issue relates to HBASE-2141 [ HBASE-2141 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1992 [ HBASE-1992 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1910 [ HBASE-1910 ]
          Hide
          ryan rawson added a comment -

          It might be better to use a framework that supports both nio and nio2. The problem with nio2 is it is java7 specific, and not released yet.

          If we want to do something like avro rpc, we should wrap this up in there.

          Show
          ryan rawson added a comment - It might be better to use a framework that supports both nio and nio2. The problem with nio2 is it is java7 specific, and not released yet. If we want to do something like avro rpc, we should wrap this up in there.
          ryan rawson made changes -
          Assignee ryan rawson [ ryanobjc ]
          Hide
          ryan rawson added a comment -

          btw im researching IO and nio frameworks now...

          Show
          ryan rawson added a comment - btw im researching IO and nio frameworks now...
          Andrew Purtell made changes -
          Summary NIO-2 based RPC NIO based RPC
          Description Reimplement HBase RPC with an NIO-2 framework. Reimplement HBase RPC with an NIO framework.
          Hide
          Andrew Purtell added a comment -

          It might be better to use a framework that supports both nio and nio2.

          Sounds good.

          Stack mentioned NIO-2 in a comment on HBASE-2141.

          Show
          Andrew Purtell added a comment - It might be better to use a framework that supports both nio and nio2. Sounds good. Stack mentioned NIO-2 in a comment on HBASE-2141 .
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1849 [ HBASE-1849 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1197 [ HBASE-1197 ]
          Andrew Purtell made changes -
          Link This issue blocks HBASE-1015 [ HBASE-1015 ]
          Andrew Purtell made changes -
          Link This issue incorporates HBASE-1863 [ HBASE-1863 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-1477 [ HBASE-1477 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-916 [ HDFS-916 ]
          Hide
          Todd Lipcon added a comment -

          I'm strongly leaning towards either Netty or no framework at all for Avro RPC and HDFS... but it's still very much work in progress. Netty currently lacks sendfile support, which is a bit of a pain.

          Show
          Todd Lipcon added a comment - I'm strongly leaning towards either Netty or no framework at all for Avro RPC and HDFS... but it's still very much work in progress. Netty currently lacks sendfile support, which is a bit of a pain.
          Hide
          stack added a comment -

          @Todd not having sendfile will be held against netty since the bulk of the data transported by DNs uses it?

          Show
          stack added a comment - @Todd not having sendfile will be held against netty since the bulk of the data transported by DNs uses it?
          Hide
          Todd Lipcon added a comment -

          Yes, but the author of Netty has said he's pretty sure he'll get sendfile done in the next couple of months. My guess is that if we ask nicely enough that could be accelerated. And the development and testing of an Netty based DN will probably take on the order of months anyhow.

          Show
          Todd Lipcon added a comment - Yes, but the author of Netty has said he's pretty sure he'll get sendfile done in the next couple of months. My guess is that if we ask nicely enough that could be accelerated. And the development and testing of an Netty based DN will probably take on the order of months anyhow.
          Hide
          stack added a comment -

          Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.

          Show
          stack added a comment - Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.
          stack made changes -
          Fix Version/s 0.22.0 [ 12314223 ]
          Fix Version/s 0.21.0 [ 12313607 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to AVRO-405 [ AVRO-405 ]
          Hide
          Andrew Purtell added a comment -

          The Netty-based Java RPC server for Avro (AVRO-405) was just committed.

          Show
          Andrew Purtell added a comment - The Netty-based Java RPC server for Avro ( AVRO-405 ) was just committed.
          Andrew Purtell made changes -
          Link This issue is blocked by AVRO-406 [ AVRO-406 ]
          Hide
          ryan rawson added a comment -

          our RPC is already NIO based It's how you USE it that counts not if it's NIO or not. For example the server RPC has 1 thread that sends replies to clients in a non-blocking manner if the handler was unable to send the entire response in the first non-blocking write.

          Show
          ryan rawson added a comment - our RPC is already NIO based It's how you USE it that counts not if it's NIO or not. For example the server RPC has 1 thread that sends replies to clients in a non-blocking manner if the handler was unable to send the entire response in the first non-blocking write.
          Todd Lipcon made changes -
          Component/s ipc [ 12312136 ]
          Component/s performance [ 12314193 ]
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]
          Hide
          Mubarak Seyed added a comment -

          Netty 3.2 (NETTY-293) supports sendfile() system call.

          Show
          Mubarak Seyed added a comment - Netty 3.2 ( NETTY-293 ) supports sendfile() system call.
          stack made changes -
          Link This issue is related to HBASE-2408 [ HBASE-2408 ]
          stack made changes -
          Link This issue is related to HBASE-3787 [ HBASE-3787 ]
          Hide
          stack added a comment -

          Made this an umbrella issue for all related to rpc. Changed the subject to be more generic

          Show
          stack added a comment - Made this an umbrella issue for all related to rpc. Changed the subject to be more generic
          stack made changes -
          Summary NIO based RPC rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.
          Issue Type Improvement [ 4 ] Umbrella [ 14 ]
          stack made changes -
          Link This issue relates to HBASE-3732 [ HBASE-3732 ]
          stack made changes -
          Link This issue relates to HBASE-3938 [ HBASE-3938 ]
          stack made changes -
          Link This issue relates to HBASE-4956 [ HBASE-4956 ]
          stack made changes -
          Link This issue relates to HBASE-4810 [ HBASE-4810 ]
          stack made changes -
          Link This issue relates to HBASE-3523 [ HBASE-3523 ]
          Hide
          stack added a comment -

          There is some overlap between these two issues

          Show
          stack added a comment - There is some overlap between these two issues
          stack made changes -
          Link This issue incorporates HBASE-5305 [ HBASE-5305 ]
          Hide
          Elliott Clark added a comment -

          So I took an early stab at this. It uses netty for both server and client. A region server has two different ports (right now). One for admin and one for data with a protobuf representation. All servers have a thread pool for IO (getting data in and out, parsing protobuf) and a thread pool for performing operations (Scanning, getting, deleting, etc). Thread pools are memory aware so slow queries shouldn't result in OOME's.

          All communication takes place through a wrapped protocol buffer protocol:
          [32 bit length field, protobuf data]

          This is pretty easy to parse and should allow native clients to be written very easily.

          All dependencies are injected into the constructor of NRegionServer and NMasterServer. This should allow us to test things a little bit easier. I had to make a couple of changes to the Zookeeper listeners for this.

          I envision the client having two implementations. The async client that just gives off futures like https://github.com/stumbleupon/asynchbase does currently. The second implementation will just be a wrapper for the async client that waits for all futures to complete before returning.

          Right now nothing works and nothing is really even close. I just wanted to put this up to get the ball rolling on discussion and see if people are even interested in this still.

          Show
          Elliott Clark added a comment - So I took an early stab at this. It uses netty for both server and client. A region server has two different ports (right now). One for admin and one for data with a protobuf representation. All servers have a thread pool for IO (getting data in and out, parsing protobuf) and a thread pool for performing operations (Scanning, getting, deleting, etc). Thread pools are memory aware so slow queries shouldn't result in OOME's. All communication takes place through a wrapped protocol buffer protocol: [32 bit length field, protobuf data] This is pretty easy to parse and should allow native clients to be written very easily. All dependencies are injected into the constructor of NRegionServer and NMasterServer. This should allow us to test things a little bit easier. I had to make a couple of changes to the Zookeeper listeners for this. I envision the client having two implementations. The async client that just gives off futures like https://github.com/stumbleupon/asynchbase does currently. The second implementation will just be a wrapper for the async client that waits for all futures to complete before returning. Right now nothing works and nothing is really even close. I just wanted to put this up to get the ball rolling on discussion and see if people are even interested in this still.
          Elliott Clark made changes -
          Attachment HBASE-2182-EARLY-0.patch [ 12534072 ]
          Hide
          Ted Yu added a comment -

          This project is related: http://code.google.com/p/netty-protobuf-rpc/
          There doesn't seem to be active development judging from open issues.

          Show
          Ted Yu added a comment - This project is related: http://code.google.com/p/netty-protobuf-rpc/ There doesn't seem to be active development judging from open issues.
          Hide
          Cosmin Lehene added a comment -

          Is this still blocked by AVRO-406?
          It looks outdated.

          Show
          Cosmin Lehene added a comment - Is this still blocked by AVRO-406 ? It looks outdated.
          stack made changes -
          Link This issue is blocked by AVRO-406 [ AVRO-406 ]
          stack made changes -
          Link This issue blocks HBASE-1015 [ HBASE-1015 ]
          Hide
          Enis Soztutar added a comment -

          No, we decided to go with PB based RPC instead of Avro RPC.

          Show
          Enis Soztutar added a comment - No, we decided to go with PB based RPC instead of Avro RPC.
          Hide
          stack added a comment -

          I removed AVRO-406 as a blocker (we don't do avro). I also removed this as a blocker on c client because Elliott seems to be making progress in spite of this issue not being done.

          We could close this issue as stale because the world has changed since this was originally written only we want the original objective, an async core, still, years later, so leaving it open for now.

          Show
          stack added a comment - I removed AVRO-406 as a blocker (we don't do avro). I also removed this as a blocker on c client because Elliott seems to be making progress in spite of this issue not being done. We could close this issue as stale because the world has changed since this was originally written only we want the original objective, an async core, still, years later, so leaving it open for now.
          Hide
          Andrew Purtell added a comment -

          This issue isn't going to happen as is. Various later issues have touched on some of this. We should create new issues with specific reasonably scoped objectives if moving this forward.

          Show
          Andrew Purtell added a comment - This issue isn't going to happen as is. Various later issues have touched on some of this. We should create new issues with specific reasonably scoped objectives if moving this forward.
          Andrew Purtell made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee ryan rawson [ ryanobjc ]
          Resolution Incomplete [ 4 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          1557d 20h 59m 1 Andrew Purtell 12/May/14 01:46

            People

            • Assignee:
              Unassigned
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development