[HBASE-18083] Make large/small file clean thread number configurable in HFileCleaner

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Hadoop Flags: Reviewed

Release Note

After HBASE-18083 we could configure HFileCleaner to use multiple threads for large/small (archived) hfile cleaning with hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count, both default to 1. These properties support online configuration change.

• Assignee: Yu Li
• Reporter: Yu Li
• Created: 19/May/17 18:48
• Updated: 14/Dec/18 10:09
• Resolved: 07/Jul/17 06:14

Description

Currently we have only one thread for both large and small file cleaning, but when write pressure is huge we might need more cleaner threads, so we need to make the thread number configurable.

We observed more than 1.8PB data in archive directory online due to business access rate change, and this proposal is one of the necessary changes required.

Default value of the configurations would still be left to 1 to keep low pressure to NN for normal case.

Issue Links

• is related to: HBASE-18309 Support multi threads in CleanerChore
• relates to: HBASE-18084 Improve CleanerChore to clean from directory which consumes more disk space