[HBASE-17757] Unify blocksize after encoding to decrease memory fragment

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0, 2.0.0
• Hadoop Flags: Reviewed

Release Note

Blocksize is set in columnfamily's atrributes. It is used to control block sizes when generating blocks. But, it doesn't take encoding into count. If you set encoding to blocks, after encoding, the block size varies. Since blocks will be cached in memory after encoding (default), it will cause memory fragment if using blockcache, or decrease the pool efficiency if using bucketCache. This issue introduced a new config named 'hbase.writer.unified.encoded.blocksize.ratio'. The default value of this config is 1, meaning doing nothing. If this value is set to a smaller value like 0.5, and the blocksize is set to 64KB(default value of blocksize). It will unify the blocksize after encoding to 64KB * 0.5 = 32KB. Unified blocksize will releaf the memory problems mentioned above.

• Assignee: Allan Yang
• Reporter: Allan Yang
• Created: 08/Mar/17 07:14
• Updated: 04/Jun/17 09:53
• Resolved: 02/May/17 04:40

Description

Usually, we store encoded block(uncompressed) in blockcache/bucketCache. Though we have set the blocksize, after encoding, blocksize is varied. Varied blocksize will cause memory fragment problem, which will result in more FGC finally.In order to relief the memory fragment, This issue adjusts the encoded block to a unified size.

Issue Links

• is related to: HBASE-18156 Provide a tool to show block cache summary