[HBASE-18309] Support multi threads in CleanerChore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 2.0.0-beta-1, 2.0.0
• Hadoop Flags: Reviewed

Release Note

After HBASE-18309,
1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir.
2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default.
3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does.
4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.

• Assignee: Reid Chan
• Reporter: Lijin Bin
• Created: 03/Jul/17 08:43
• Updated: 16/Aug/19 10:49
• Resolved: 05/Apr/18 06:02

Description

There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash not happened any more.

What's more, currently there's only one thread iterating the archive directory, and we could use multiple threads cleaning sub directories in parallel to speed it up.

Issue Links

• causes: HBASE-22867 The ForkJoinPool in CleanerChore will spawn thousands of threads in our cluster with thousands table
• is depended upon by: HBASE-19306 Avoid thread flood when both LogCleaner and HFileCleaner support multi threads execution
• is depended upon by: HBASE-20352 [Chore] Backport HBASE-18309 to branch-1
• is related to: HBASE-20401 Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
• relates to: HBASE-19709 Guard against a ThreadPool size of 0 in CleanerChore
• relates to: HBASE-20095 Redesign single instance pool in CleanerChore
• relates to: HBASE-18083 Make large/small file clean thread number configurable in HFileCleaner
• relates to: HBASE-14247 Separate the old WALs into different regionserver directories
• links to: Review Board