Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15555 Memory Management
  3. HBASE-15525

OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.0.0
    • IPC/RPC
    • None
    • Reviewed
    • Hide
      Added a new ByteBufferPool which pools N ByteBuffers. By default it makes off heap ByteBuffers when getBuffer() is called. The size of each buffer defaults to 64KB. This can be configured using 'hbase.ipc.server.reservoir.initial.buffer.size'. The max number of buffers which can be pooled defaults to twice the number of handler threads in RS. This can be configured with key 'hbase.ipc.server.reservoir.initial.max'. While responding to read requests and client support Codec, we will create CellBlocks and directly return it as PB payload. For making this block, we will use N ByteBuffers from pool as per the total size of the response cells. The default size of 64 KB for the buffer is inline with the number of bytes written to RPC layer in one short.(That is also 64KB). When at point of time, the calle not able to get a free buffer from the pool (it returns null then), it will make on heap Buffer of same size (as that of Buffers in pool) and use that to create cell block.
      Show
      Added a new ByteBufferPool which pools N ByteBuffers. By default it makes off heap ByteBuffers when getBuffer() is called. The size of each buffer defaults to 64KB. This can be configured using 'hbase.ipc.server.reservoir.initial.buffer.size'. The max number of buffers which can be pooled defaults to twice the number of handler threads in RS. This can be configured with key 'hbase.ipc.server.reservoir.initial.max'. While responding to read requests and client support Codec, we will create CellBlocks and directly return it as PB payload. For making this block, we will use N ByteBuffers from pool as per the total size of the response cells. The default size of 64 KB for the buffer is inline with the number of bytes written to RPC layer in one short.(That is also 64KB). When at point of time, the calle not able to get a free buffer from the pool (it returns null then), it will make on heap Buffer of same size (as that of Buffers in pool) and use that to create cell block.

    Description

      After HBASE-13819 the system some times run out of direct memory whenever there is some network congestion or some client side issues.
      This was because of pending RPCs in the RPCServer$Connection.responseQueue and since all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards a higher value

      See the discussion here HBASE-13819-comment

      Attachments

        1. WIP.patch
          29 kB
          Anoop Sam John
        2. HBASE-15525_WIP.patch
          31 kB
          Anoop Sam John
        3. HBASE-15525_V1.patch
          35 kB
          Anoop Sam John
        4. HBASE-15525_V2.patch
          36 kB
          Anoop Sam John
        5. HBASE-15525_V3.patch
          36 kB
          Anoop Sam John
        6. HBASE-15525_V4.patch
          36 kB
          Anoop Sam John
        7. HBASE-15525_V5.patch
          37 kB
          Anoop Sam John
        8. HBASE-15525_V6.patch
          40 kB
          Anoop Sam John

        Issue Links

          Activity

            People

              anoop.hbase Anoop Sam John
              dvdreddy deepankar
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: