[HBASE-8329] Limit compaction speed

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0, 0.98.11, 2.0.0
• Component/s: Compaction
• Hadoop Flags: Reviewed

Release Note

Adds compaction throughput limit mechanism(the word "throttle" is already used when choosing compaction thread pool, so use a different word here to avoid ambiguity). Default is org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompactionThroughputController, will limit throughput as follow:
1. In off peak hours, use a fixed limitation "hbase.hstore.compaction.throughput.offpeak" (default is Long.MAX_VALUE which means no limitation).
2. In normal hours, the limitation is tuned between "hbase.hstore.compaction.throughput.lower.bound"(default 10MB/sec) and "hbase.hstore.compaction.throughput.higher.bound"(default 20MB/sec), using the formula "lower + (higer - lower) * param" where param is in range [0.0, 1.0] and calculate based on store files count on this regionserver.
3. If some stores have too many store files(storefilesCount > blockingFileCount), then there is no limitation no matter peak or off peak.
You can set "hbase.regionserver.throughput.controller" to org.apache.hadoop.hbase.regionserver.throttle.NoLimitThroughputController to disable throughput controlling.
And we have implemented ConfigurationObserver which means you can change all configurations above and do not need to restart cluster.

The throttle is on by default in hbase-2.0.0. There is no limit in hbase-1.x.

• Assignee: Duo Zhang
• Reporter: Lijin Bin
• Created: 12/Apr/13 02:27
• Updated: 28/Dec/17 16:28
• Resolved: 03/Feb/15 06:21

Description

There is no speed or resource limit for compaction，I think we should add this feature especially when request burst.

Issue Links

• is duplicated by: HBASE-3743 Throttle major compaction
• relates to: HBASE-12986 Compaction pressure based client pushback
• links to: reviewboard

Sub-Tasks

• DEPRECATED_NAME_OF_NO_LIMIT_THROUGHPUT_CONTROLLER_CLASS value is wrong in CompactionThroughputControllerFactory: Resolved