[HBASE-16213] A new HFileBlock structure for fast random get

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 2.0.0
• Component/s: Performance
• Hadoop Flags: Reviewed

Release Note

HBASE-16213 introduced a new DataBlockEncoding in name of ROW_INDEX_V1, which could improve random read (get) performance especially when the average record size (key-value size per row) is small. To use this feature, please set DATA_BLOCK_ENCODING to ROW_INDEX_V1 for CF of newly created table, or change existing CF with below command:
alter 'table_name',{NAME => 'cf', DATA_BLOCK_ENCODING => 'ROW_INDEX_V1'}.

Please note that if we turn this DBE on, HFile block will be bigger than NONE encoding because it adds some meta infos for binary search:
/**
 * Store cells following every row's start offset, so we can binary search to a row's cells.
 *
 * Format:
 * flat cells
 * integer: number of rows
 * integer: row0's offset
 * integer: row1's offset
 * ....
 * integer: dataSize
 *
*/

Seek in row when random reading is one of the main consumers of CPU. This helps. See slide #7 here https://www.slideshare.net/HBaseCon/lift-the-ceiling-of-hbase-throughputs?qid=597ee2fa-8125-4faa-bb3b-2bf1ba9ccafb&v=&b=&from_search=6

• Assignee: Lijin Bin
• Reporter: Lijin Bin
• Created: 12/Jul/16 05:12
• Updated: 02/Oct/19 23:25
• Resolved: 29/Aug/16 06:42

Description

HFileBlock store cells sequential, current when to get a row from the block, it scan from the first cell until the row's cell.
The new structure store every row's start offset with data, so it can find the exact row with binarySearch.

I use EncodedSeekPerformanceTest test the performance.
First use ycsb write 100w data, every row have only one qualifier, and valueLength=16B/64/256B/1k.
Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and also record HFileBlock's dataSize/dataWithMetaSize in the encoding.

Issue Links

• is related to: HBASE-21401 Sanity check when constructing the KeyValue
• is related to: HBASE-21473 RowIndexSeekerV1 may return cell with extra two \x00\x00 bytes which has no tags
• relates to: HBASE-16502 Reduce garbage in BufferedDataBlockEncoder
• links to: ReviewBoard Link

Sub-Tasks

• Reduce DBE code duplication: Resolved