[HBASE-1476] scaling compaction with multiple threads

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.92.0
• Component/s: regionserver
• Labels: moved_from_0_20_5
• Assignee: Nicolas Spiegelberg
• Reporter: Billy Pearson
• Created: 02/Jun/09 18:34
• Updated: 20/Nov/15 13:01
• Resolved: 10/May/11 23:37

Description

Was thinking we should build in support to be able to handle more then one thread for compactions this will allow us to keep up with compactions when we get to the point where we store Tb's of data per node and may regions
Maybe a configurable setting to set how many threads a region server can use for compactions.

With compression turned on my compactions are limited by cpu speed with multi cores then it would be nice to be able to scale compactions to 2 or more cores.

Issue Links

• blocks: HBASE-1845 MultiGet, MultiDelete, and MultiPut - batched to the appropriate region servers
• blocks: HBASE-1306 [performance] Threading the update process when writing to HBase.
• depends upon: HBASE-3797 StoreFile Level Compaction Locking
• is required by: HBASE-3877 Determine Proper Defaults for Compaction ThreadPools
• relates to: HBASE-2832 Priorities and multi-threading for MemStore flushing