Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15555 Memory Management
  3. HBASE-15525

OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.0.0
    • IPC/RPC
    • None
    • Reviewed
    • Hide
      Added a new ByteBufferPool which pools N ByteBuffers. By default it makes off heap ByteBuffers when getBuffer() is called. The size of each buffer defaults to 64KB. This can be configured using 'hbase.ipc.server.reservoir.initial.buffer.size'. The max number of buffers which can be pooled defaults to twice the number of handler threads in RS. This can be configured with key 'hbase.ipc.server.reservoir.initial.max'. While responding to read requests and client support Codec, we will create CellBlocks and directly return it as PB payload. For making this block, we will use N ByteBuffers from pool as per the total size of the response cells. The default size of 64 KB for the buffer is inline with the number of bytes written to RPC layer in one short.(That is also 64KB). When at point of time, the calle not able to get a free buffer from the pool (it returns null then), it will make on heap Buffer of same size (as that of Buffers in pool) and use that to create cell block.
      Show
      Added a new ByteBufferPool which pools N ByteBuffers. By default it makes off heap ByteBuffers when getBuffer() is called. The size of each buffer defaults to 64KB. This can be configured using 'hbase.ipc.server.reservoir.initial.buffer.size'. The max number of buffers which can be pooled defaults to twice the number of handler threads in RS. This can be configured with key 'hbase.ipc.server.reservoir.initial.max'. While responding to read requests and client support Codec, we will create CellBlocks and directly return it as PB payload. For making this block, we will use N ByteBuffers from pool as per the total size of the response cells. The default size of 64 KB for the buffer is inline with the number of bytes written to RPC layer in one short.(That is also 64KB). When at point of time, the calle not able to get a free buffer from the pool (it returns null then), it will make on heap Buffer of same size (as that of Buffers in pool) and use that to create cell block.

    Description

      After HBASE-13819 the system some times run out of direct memory whenever there is some network congestion or some client side issues.
      This was because of pending RPCs in the RPCServer$Connection.responseQueue and since all the responses in this queue hold a buffer for cellblock from BoundedByteBufferPool this could takeup a lot of memory if the BoundedByteBufferPool's moving average settles down towards a higher value

      See the discussion here HBASE-13819-comment

      Attachments

        1. HBASE-15525_V1.patch
          35 kB
          Anoop Sam John
        2. HBASE-15525_V2.patch
          36 kB
          Anoop Sam John
        3. HBASE-15525_V3.patch
          36 kB
          Anoop Sam John
        4. HBASE-15525_V4.patch
          36 kB
          Anoop Sam John
        5. HBASE-15525_V5.patch
          37 kB
          Anoop Sam John
        6. HBASE-15525_V6.patch
          40 kB
          Anoop Sam John
        7. HBASE-15525_WIP.patch
          31 kB
          Anoop Sam John
        8. WIP.patch
          29 kB
          Anoop Sam John

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HBASE-13819 Make RPC layer CellBlock buffer a DirectByteBuffer

          • Major - Major loss of function.
          • Closed

          Umbrella - An overarching type made of sub-tasks HBASE-11425 Cell/DBB end-to-end on the read-path

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-17073 Increase the max number of buffers in ByteBufferPool

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HBASE-17805 We should remove BoundedByteBufferPool because it is replaced by ByteBufferPool

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              anoop.hbase Anoop Sam John
              dvdreddy deepankar
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: