[HBASE-16417] In-Memory MemStore Policy for Flattening and Compactions

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 2.0.0

Release Note

A new memstore compaction policy: Adaptive.
Adaptive compaction adapts to the workload. It applies either index compaction or data compaction based on the ratio of redundant cells.

New configuration properties:
(1) hbase.hregion.compacting.memstore.adaptive.compaction.threshold
sets the threshold on the fraction of redundant cells to trigger memstore data compaction.
Default value is 0.5.

(2) hbase.hregion.compacting.memstore.adaptive.compaction.probability
sets the initial probability of applying data compaction (a throttling parameter). Default value is 0.5.

In addition
- changes the default value of the configuration property
hbase.memstore.inmemoryflush.threshold.factor to 0.02 (previously 0.25)
 - changes the default value of the configuration property hbase.hregion.compacting.pipeline.segments.limit to 4 (previously 1)

• Assignee: Eshcar Hillel
• Reporter: Anastasia Braginsky
• Created: 15/Aug/16 10:25
• Updated: 24/Jun/22 18:15
• Resolved: 02/Nov/17 12:11

Description

This Jira explores the performance of different memstore compaction policies.
It presents the result of write-only workload evaluation as well as read performance in read-write workloads.
We investigate several settings of hardware (SSD, HDD), key distribution (Zipf, uniform), with multiple settings of the system, and compare measures like write throughput, read latency, write volume, total gc time, etc.

The submitted patch sets some system properties at the values yielding optimal performance. In addition we suggest a new Adaptive memstore compaction policy that shows good tradeoffs between write throughput and write volume.

Issue Links

• relates to: HBASE-19161 For system tables, if config 'hbase.systemtables.compacting.memstore.type' is not specified then DefaultMemStore is used
• links to: Review board