[HBASE-21657] PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0
• Component/s: Performance
• Hadoop Flags: Incompatible change, Reviewed

Release Note

In HBASE-21657, I simplified the path of estimatedSerialiedSize() & estimatedSerialiedSizeOfCell() by moving the general getSerializedSize()
and heapSize() from ExtendedCell to Cell interface. The patch also included some other improvments:

1. For 99% of case, our cells has no tags, so let the HFileScannerImpl just return the NoTagsByteBufferKeyValue if no tags, which means we can save
   lots of cpu time when sending no tags cell to rpc because can just return the length instead of getting the serialize size by caculating offset/length
   of each fields(row/cf/cq..)
2. Move the subclass's getSerializedSize implementation from ExtendedCell to their own class, which mean we did not need to call ExtendedCell's
   getSerialiedSize() firstly, then forward to subclass's getSerializedSize(withTags).
3. Give a estimated result arraylist size for avoiding the frequent list extension when in a big scan, now we estimate the array size as min(scan.rows, 512).
   it's also help a lot.

We gain almost ~40% throughput improvement in 100% scan case for branch-2 (cacheHitRatio~100%)[1], it's a good thing. While it's a incompatible change in
some case, such as if the upstream user implemented their own Cells, although it's rare but can happen, then their compile will be error.

• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 28/Dec/18 09:11
• Updated: 16/Jan/19 07:49
• Resolved: 16/Jan/19 02:11

Description

We are evaluating the performance of branch-2, and find that the throughput of scan in SSD cluster is almost the same as HDD cluster. so I made a FlameGraph on RS, and found that the PrivateCellUtil#estimatedSerializedSizeOf cost about 29% cpu, Obviously, it has been the bottleneck in 100% scan case.

See the hbase20-ssd-100-scan-traces.svg

BTW, in our XiaoMi branch, we introduce a HRegion#updateReadRequestsByCapacityUnitPerSecond to sum up the size of cells (for metric monitor), so it seems the performance loss was amplified.

Issue Links

• relates to: HBASE-21616 Port HBASE-21034 (Add new throttle type: read/write capacity unit) to branch-1
• relates to: HBASE-13307 Making methods under ScannerV2#next inlineable, faster
• relates to: HBASE-20188 Evaluate and address performance delta between branch-1 and branch-2
• links to: Review board