Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Resolved
-
None
-
None
-
None
-
Reviewed
-
Description
There is only one thread in LogCleaner to clean oldWALs and in our big cluster we find this is not enough. The number of files under oldWALs reach the max-directory-items limit of HDFS and cause region server crash, so we use multi threads for LogCleaner and the crash not happened any more.
What's more, currently there's only one thread iterating the archive directory, and we could use multiple threads cleaning sub directories in parallel to speed it up.
Attachments
Attachments
Issue Links
- causes
-
HBASE-22867 The ForkJoinPool in CleanerChore will spawn thousands of threads in our cluster with thousands table
-
- Resolved
-
- is depended upon by
-
HBASE-19306 Avoid thread flood when both LogCleaner and HFileCleaner support multi threads execution
-
- Open
-
-
HBASE-20352 [Chore] Backport HBASE-18309 to branch-1
-
- Resolved
-
- is related to
-
HBASE-20401 Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
-
- Resolved
-
- relates to
-
HBASE-19709 Guard against a ThreadPool size of 0 in CleanerChore
-
- Resolved
-
-
HBASE-20095 Redesign single instance pool in CleanerChore
-
- Resolved
-
-
HBASE-18083 Make large/small file clean thread number configurable in HFileCleaner
-
- Resolved
-
-
HBASE-14247 Separate the old WALs into different regionserver directories
-
- Closed
-
- links to