HBase
  1. HBase
  2. HBASE-10374

add jemalloc into the choice of memstore allocation

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.96.1.1
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None

      Description

      https://blog.twitter.com/2013/netty-4-at-twitter-reduced-gc-overhead introduced that Netty used jemalloc to gain benefits from GC.
      It can be a good choice for memstore block allocation.

        Activity

        Hide
        stack added a comment -

        Bing Jiang What you thinking? We pull in netty4, redo our core on top of it (smile), then as a side benefit, use their jemalloc implementation for block cache? Poking around, I don't see it here http://netty.io/4.0/api/index.html Thanks.

        Show
        stack added a comment - Bing Jiang What you thinking? We pull in netty4, redo our core on top of it (smile), then as a side benefit, use their jemalloc implementation for block cache? Poking around, I don't see it here http://netty.io/4.0/api/index.html Thanks.
        Show
        Andrew Purtell added a comment - I found it here: http://netty.io/4.0/api/io/netty/buffer/ByteBufAllocator.html
        Hide
        Andrew Purtell added a comment -

        Someone should try pulling in netty4 while excluding earlier versions to see what happens. I bet it will be a mess given our dependencies on Jetty 6 and ZooKeeper.

        Show
        Andrew Purtell added a comment - Someone should try pulling in netty4 while excluding earlier versions to see what happens. I bet it will be a mess given our dependencies on Jetty 6 and ZooKeeper.
        Hide
        Nick Dimiduk added a comment -

        Yeah, I tried this once and it didn't go very well. I am keen to pursue Stack's suggestion though. I haven't come to a conclusive decision for why we'd pursue Netty's ByteBuf vs nio

        Show
        Nick Dimiduk added a comment - Yeah, I tried this once and it didn't go very well. I am keen to pursue Stack's suggestion though. I haven't come to a conclusive decision for why we'd pursue Netty's ByteBuf vs nio
        Hide
        Nick Dimiduk added a comment -

        Premature submit. As I was a saying.

        I don't have conclusive reasons to pursue Netty ByteBufs vs nio ByteBuffers. The compositional structure Netty provides is really nice, if we can use it, but I think it could be implemented on top of stock ByteBuffers as well.

        Show
        Nick Dimiduk added a comment - Premature submit. As I was a saying. I don't have conclusive reasons to pursue Netty ByteBufs vs nio ByteBuffers. The compositional structure Netty provides is really nice, if we can use it, but I think it could be implemented on top of stock ByteBuffers as well.
        Hide
        Andrew Purtell added a comment -

        This is also directly related to HBASE-10191

        Show
        Andrew Purtell added a comment - This is also directly related to HBASE-10191
        Hide
        Bing Jiang added a comment -

        stack I have a question, if we try to use the feature which Netty provides, I believe there are two choices, the one is to add dependence upon Netty, the other one is to implements the jemalloc by the help of their codes. I want to take your suggestion.

        Show
        Bing Jiang added a comment - stack I have a question, if we try to use the feature which Netty provides, I believe there are two choices, the one is to add dependence upon Netty, the other one is to implements the jemalloc by the help of their codes. I want to take your suggestion.
        Hide
        stack added a comment -

        Bing Jiang You suggest jemalloc for memstore block allocations. What are you thinking we'll see by way of benefit? I'd think that isolating the MemStore and then in a simple testing harness trying various options would be the way to go (MemStore can be stood up outside of a HStore IIRC). Try netty implementation first since code is done. See if you can get any speedup in your testing rig. If improvement, then lets talk. We'll have to see about what Nick Dimiduk reminds us of, that netty implemenation is ByteBuf-based (as opposed to ByteBuffer).

        On pulling in netty4, it might not be too bad since they changed the package from org.jboss to io.netty.

        Show
        stack added a comment - Bing Jiang You suggest jemalloc for memstore block allocations. What are you thinking we'll see by way of benefit? I'd think that isolating the MemStore and then in a simple testing harness trying various options would be the way to go (MemStore can be stood up outside of a HStore IIRC). Try netty implementation first since code is done. See if you can get any speedup in your testing rig. If improvement, then lets talk. We'll have to see about what Nick Dimiduk reminds us of, that netty implemenation is ByteBuf-based (as opposed to ByteBuffer). On pulling in netty4, it might not be too bad since they changed the package from org.jboss to io.netty.

          People

          • Assignee:
            Unassigned
            Reporter:
            Bing Jiang
          • Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development