HBase
  1. HBase
  2. HBASE-2182

rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.

    Details

    • Type: Umbrella Umbrella
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: IPC/RPC, Performance
    • Labels:
      None

      Description

      Reimplement HBase RPC with an NIO framework.

        Issue Links

          Activity

          Hide
          stack added a comment -

          I removed AVRO-406 as a blocker (we don't do avro). I also removed this as a blocker on c client because Elliott seems to be making progress in spite of this issue not being done.

          We could close this issue as stale because the world has changed since this was originally written only we want the original objective, an async core, still, years later, so leaving it open for now.

          Show
          stack added a comment - I removed AVRO-406 as a blocker (we don't do avro). I also removed this as a blocker on c client because Elliott seems to be making progress in spite of this issue not being done. We could close this issue as stale because the world has changed since this was originally written only we want the original objective, an async core, still, years later, so leaving it open for now.
          Hide
          Enis Soztutar added a comment -

          No, we decided to go with PB based RPC instead of Avro RPC.

          Show
          Enis Soztutar added a comment - No, we decided to go with PB based RPC instead of Avro RPC.
          Hide
          Cosmin Lehene added a comment -

          Is this still blocked by AVRO-406?
          It looks outdated.

          Show
          Cosmin Lehene added a comment - Is this still blocked by AVRO-406 ? It looks outdated.
          Hide
          Ted Yu added a comment -

          This project is related: http://code.google.com/p/netty-protobuf-rpc/
          There doesn't seem to be active development judging from open issues.

          Show
          Ted Yu added a comment - This project is related: http://code.google.com/p/netty-protobuf-rpc/ There doesn't seem to be active development judging from open issues.
          Hide
          Elliott Clark added a comment -

          So I took an early stab at this. It uses netty for both server and client. A region server has two different ports (right now). One for admin and one for data with a protobuf representation. All servers have a thread pool for IO (getting data in and out, parsing protobuf) and a thread pool for performing operations (Scanning, getting, deleting, etc). Thread pools are memory aware so slow queries shouldn't result in OOME's.

          All communication takes place through a wrapped protocol buffer protocol:
          [32 bit length field, protobuf data]

          This is pretty easy to parse and should allow native clients to be written very easily.

          All dependencies are injected into the constructor of NRegionServer and NMasterServer. This should allow us to test things a little bit easier. I had to make a couple of changes to the Zookeeper listeners for this.

          I envision the client having two implementations. The async client that just gives off futures like https://github.com/stumbleupon/asynchbase does currently. The second implementation will just be a wrapper for the async client that waits for all futures to complete before returning.

          Right now nothing works and nothing is really even close. I just wanted to put this up to get the ball rolling on discussion and see if people are even interested in this still.

          Show
          Elliott Clark added a comment - So I took an early stab at this. It uses netty for both server and client. A region server has two different ports (right now). One for admin and one for data with a protobuf representation. All servers have a thread pool for IO (getting data in and out, parsing protobuf) and a thread pool for performing operations (Scanning, getting, deleting, etc). Thread pools are memory aware so slow queries shouldn't result in OOME's. All communication takes place through a wrapped protocol buffer protocol: [32 bit length field, protobuf data] This is pretty easy to parse and should allow native clients to be written very easily. All dependencies are injected into the constructor of NRegionServer and NMasterServer. This should allow us to test things a little bit easier. I had to make a couple of changes to the Zookeeper listeners for this. I envision the client having two implementations. The async client that just gives off futures like https://github.com/stumbleupon/asynchbase does currently. The second implementation will just be a wrapper for the async client that waits for all futures to complete before returning. Right now nothing works and nothing is really even close. I just wanted to put this up to get the ball rolling on discussion and see if people are even interested in this still.
          Hide
          stack added a comment -

          There is some overlap between these two issues

          Show
          stack added a comment - There is some overlap between these two issues
          Hide
          stack added a comment -

          Made this an umbrella issue for all related to rpc. Changed the subject to be more generic

          Show
          stack added a comment - Made this an umbrella issue for all related to rpc. Changed the subject to be more generic
          Hide
          Mubarak Seyed added a comment -

          Netty 3.2 (NETTY-293) supports sendfile() system call.

          Show
          Mubarak Seyed added a comment - Netty 3.2 ( NETTY-293 ) supports sendfile() system call.
          Hide
          stack added a comment -

          Moving out of 0.92.0. Pull it back in if you think different.

          Show
          stack added a comment - Moving out of 0.92.0. Pull it back in if you think different.
          Hide
          ryan rawson added a comment -

          our RPC is already NIO based It's how you USE it that counts not if it's NIO or not. For example the server RPC has 1 thread that sends replies to clients in a non-blocking manner if the handler was unable to send the entire response in the first non-blocking write.

          Show
          ryan rawson added a comment - our RPC is already NIO based It's how you USE it that counts not if it's NIO or not. For example the server RPC has 1 thread that sends replies to clients in a non-blocking manner if the handler was unable to send the entire response in the first non-blocking write.
          Hide
          Andrew Purtell added a comment -

          The Netty-based Java RPC server for Avro (AVRO-405) was just committed.

          Show
          Andrew Purtell added a comment - The Netty-based Java RPC server for Avro ( AVRO-405 ) was just committed.
          Hide
          stack added a comment -

          Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.

          Show
          stack added a comment - Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.
          Hide
          Todd Lipcon added a comment -

          Yes, but the author of Netty has said he's pretty sure he'll get sendfile done in the next couple of months. My guess is that if we ask nicely enough that could be accelerated. And the development and testing of an Netty based DN will probably take on the order of months anyhow.

          Show
          Todd Lipcon added a comment - Yes, but the author of Netty has said he's pretty sure he'll get sendfile done in the next couple of months. My guess is that if we ask nicely enough that could be accelerated. And the development and testing of an Netty based DN will probably take on the order of months anyhow.
          Hide
          stack added a comment -

          @Todd not having sendfile will be held against netty since the bulk of the data transported by DNs uses it?

          Show
          stack added a comment - @Todd not having sendfile will be held against netty since the bulk of the data transported by DNs uses it?
          Hide
          Todd Lipcon added a comment -

          I'm strongly leaning towards either Netty or no framework at all for Avro RPC and HDFS... but it's still very much work in progress. Netty currently lacks sendfile support, which is a bit of a pain.

          Show
          Todd Lipcon added a comment - I'm strongly leaning towards either Netty or no framework at all for Avro RPC and HDFS... but it's still very much work in progress. Netty currently lacks sendfile support, which is a bit of a pain.
          Hide
          Andrew Purtell added a comment -

          It might be better to use a framework that supports both nio and nio2.

          Sounds good.

          Stack mentioned NIO-2 in a comment on HBASE-2141.

          Show
          Andrew Purtell added a comment - It might be better to use a framework that supports both nio and nio2. Sounds good. Stack mentioned NIO-2 in a comment on HBASE-2141 .
          Hide
          ryan rawson added a comment -

          btw im researching IO and nio frameworks now...

          Show
          ryan rawson added a comment - btw im researching IO and nio frameworks now...
          Hide
          ryan rawson added a comment -

          It might be better to use a framework that supports both nio and nio2. The problem with nio2 is it is java7 specific, and not released yet.

          If we want to do something like avro rpc, we should wrap this up in there.

          Show
          ryan rawson added a comment - It might be better to use a framework that supports both nio and nio2. The problem with nio2 is it is java7 specific, and not released yet. If we want to do something like avro rpc, we should wrap this up in there.

            People

            • Assignee:
              ryan rawson
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:

                Development