[HBASE-5349] Automagically tweak global memstore and block cache sizes based on workload

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.92.0
• Fix Version/s: 0.99.0
• Hadoop Flags: Reviewed

Release Note

Both memstore heap size and on heap block cache sizes can be tuned automatically within the RS life time. The algorithm to be used for this tuning is also pluggable.
This feature adds the below new config
"hbase.regionserver.global.memstore.size.max.range" and "hbase.regionserver.global.memstore.size.min.range" using which one can specify the total heap % within which the memstore size can vary.
"hfile.block.cache.size.max.range" and "hfile.block.cache.size.min.range" using which one can specify the total heap % within which the block cache size can vary.
Using "hbase.regionserver.heapmemory.tuner.class" one can plugin an impl for the tuner algorithm. Pass the FQCN of the tuner impl class which implements org.apache.hadoop.hbase.regionserver.HeapMemoryTuner
The period within which the tuner checks for a possible tune can be adjusted with "hbase.regionserver.heapmemory.tuner.period". This defaults to 300000 (5 mins)
The tuner algorithm receives a TunerContext in which we pass the number of block cache evictions (happened within that time) and the memstore flushes happened (Forced due to global heap pressure and normal flushes as separate items)

Config changes
------------------
Changed the config name "hbase.regionserver.global.memstore.upperLimit" to "hbase.regionserver.global.memstore.size"
This will be the initial memstore size allocated (default value as 40%).
Also changed "hbase.regionserver.global.memstore.lowerLimit" to "hbase.regionserver.global.memstore.size.lower.limit"
There is a semantic change also here. "hbase.regionserver.global.memstore.lowerLimit" represented the total heap % upon which we start force flushes for memstores and defaults to 38% of total heap space.
The new one is the % of memstore max heap size (Represented by hbase.regionserver.global.memstore.size). This defaults to 95% of hbase.regionserver.global.memstore.size value.

• Assignee: Anoop Sam John
• Reporter: Jean-Daniel Cryans
• Created: 07/Feb/12 22:36
• Updated: 21/Feb/15 23:35
• Resolved: 11/Dec/13 07:05

Description

Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/

That'd be one less thing to configure.

Issue Links

• is related to: HBASE-2706 Use heuristics to adjust various parameters based on load characteristics