[HBASE-23598] There are too much small WAL File

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.6, 2.2.2
• Component/s: wal

Environment

hbase version: cdh5-1.2.0_5.14.4

hbase.wal.provider: multiwal

hbase.wal.regiongrouping.numgroups: 4

The wals file shows 100+ wal files in wal-3 , and some of them has very small size

• Assignee: Unassigned
• Reporter: zhuobin zheng
• Created: 19/Dec/19 12:19
• Updated: 19/Dec/19 13:47

Description

I found 10W + WAL files in my 400-scale hbase cluster. Too many WAL files will cause the cluster and recover very slowly when cluster crash completely . (In the split log step) (because too many WAL files will cause too many ZK requests). By default, WAL files start to roll when they reach HDFS Block Size (256M In My Case) * 0.95. But I found that there are many small files (0-100M) in the WAL directory. When I look at the code , I found that when I configured multiwal (I configured 4 WALs for each RS), as long as a single WAL file reached HDFS Block Size (256M In My Case) * 0.95, all WAL files would scroll, so it caused a lot of WAL small files.
I tried to modify the code to solve the problem (making each WAL scroll independently). Although this change is very small, I am not sure if such a change will cause other problems, currently being tested ...