[HBASE-13142] [PERF] Reuse the IPCUtil#buildCellBlock buffer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0, 2.0.0
• Component/s: Performance
• Labels: beginner
• Hadoop Flags: Reviewed

Release Note

Adds buffer reuse sending Cell results. It is on by default and should not need configuration. Improves GC profile and ups throughput. The benefit gets better the larger the row size returned.

The buffer reservoir is bounded at a maximum count after which we will start logging at WARN level that the reservoir is running at capacity (returned buffers will be discarded and not added back to the reservoir pool). Default maximum is twice the handler count: i.e. 2 * hbase.regionserver.handler.count. This should be more than enough. Set the maximum with the new configuration: hbase.ipc.server.reservoir.max

The reservoir will not cache buffers in excess of hbase.ipc.server.reservoir.max.buffer.size The default is 10MB. This means that if a row is very large, then we will allocate a buffer of the average size that is currently in the pool and we will then resize it till we can accommodate the return. These resizes are expensive. The resultant buffer will be used and then discarded.

To check how the reservoir is doing, enable trace level logging for a few seconds on a regionserver. You can do this from the regionserver UI. See 'Log Level'. Set org.apache.hadoop.hbase.io.BoundedByteBufferPool to TRACE. The BoundedByteBufferPool will spew report to the log. Disable the TRACE level and then check the log. You'll see allocation rate, size of pool, size of buffers in pool, etc.

• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 03/Mar/15 04:44
• Updated: 23/Dec/15 05:57
• Resolved: 06/Mar/15 22:37

Description

Running some scan profiling, flight recorder was mildly fingering resize of the buffer allocated in IPCUtil#buildCellBlock as a point of contention. It was half-hearted blaming it for a few hundreds of ms over a five minute sampling with a few tens of instances showing.

I tried then w/ flamegraph/lightweight profiler and this reported the buffer allocations taking 22% of our total CPU. See attachment trace.svg.

I enabled TRACE-level logging on org.apache.hadoop.hbase.ipc.IPCUtil and indeed every allocation was doing a resize from initial allocation of 16k – the default up to 220k (this test returns ten randomly sized rows zipfian sized between 0 and 8k).

Upping the allocation to 220k meant we now avoided the resize but the initial allocation was now blamed for 10% of allocations (see trace.2.svg attached).

Lets do buffer reuse. Will save a bunch of allocation and CPU.

Issue Links

• depends upon: HBASE-13188 java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
• relates to: HBASE-13188 java.lang.ArithmeticException issue in BoundedByteBufferPool.putBuffer
• relates to: HBASE-13778 BoundedByteBufferPool incorrectly increasing runningAverage buffer length

Sub-Tasks

• Backport to 0.98 "[PERF] Reuse the IPC buffers...": Closed