[HBASE-18309] Support multi threads in CleanerChore - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Resolved
Affects Version/s: None
Fix Version/s: 2.0.0-beta-1, 2.0.0
Component/s: None
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
After ~~HBASE-18309~~,
1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir.
2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default.
3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does.
4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.

Show
After HBASE-18309 , 1. we could use multiple threads to scan archive directories (including data and oldWALs) through config hbase.cleaner.scan.dir.concurrent.size, which supports both integer (meaning the concrete size, but no more than available cpu cores) and double (between >0.0 and <=1.0, meaning ratio of available cpu cores) value and defaults to 0.25. Pay attention that 1.0 is different from 1, the former indicates it will use 100% of cores, while the latter will use only 1 thread for chore to scan dir. 2. We also support using multiple threads to clean wals under oldWALs directory through hbase.oldwals.cleaner.thread.size, 2 by default. 3. In addition, hbase.cleaner.scan.dir.concurrent.size and hbase.oldwals.cleaner.thread.size support online re-configuration as hbase.regionserver.hfilecleaner.large.thread.count and hbase.regionserver.hfilecleaner.small.thread.count does. 4. Please take hbase.cleaner.scan.dir.concurrent.size, hbase.regionserver.hfilecleaner.large.thread.count, hbase.regionserver.hfilecleaner.small.thread.count and hbase.oldwals.cleaner.thread.size into account when setting this config to avoid thread flooding.

Description

There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash not happened any more.

What's more, currently there's only one thread iterating the archive directory, and we could use multiple threads cleaning sub directories in parallel to speed it up.