[HBASE-23340] hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldWALs too large (more than 2TB)

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1, 2.2.3
• Fix Version/s: 3.0.0-alpha-1, 2.5.0
• Component/s: master

Release Note

Previously the LogCleaner chores had their own ZK client. If they encounter Session expired error, the LogCleaner chore will never succeed again despite the HMaster continuing to run. With this change, the LogCleaner chores now share the underlying ZK of the HMaster (similar to HFileCleaner chores). So now, if an unrecoverable session expiration occurs, the hmaster will abort and cleaner chores will not be left as zombies.

• Assignee: Bo Cui
• Reporter: Jacky Lau
• Created: 25/Nov/19 08:53
• Updated: 06/Jan/23 19:47
• Resolved: 06/Jan/23 19:47

Description

hmaster /hbase/replication/rs session expired (hbase replication default value is true, we don't use ) causes logcleaner can not clean oldWALs, which resulits in oldWALs too large (more than 2TB).

 

 

we can solve it by following :

1) increase the session timeout(but i think it is not a good idea. because we do not know how long to set is suitable)

2) close the hbase replication. It is not a good idea too, when our user uses this feature

3) we need add retry times, for example when it has already happened three times, we set the ReplicationLogCleaner and SnapShotCleaner stop

that is all my ideas, i do not konw it is suitable, If it is suitable, could i commit a PR?

Does anynode have a good idea.

Issue Links

• links to: GitHub Pull Request #2689
• links to: GitHub Pull Request #2739
• links to: GitHub Pull Request #2779
• links to: GitHub Pull Request #4944

Sub-Tasks

• Backport HBASE-23340 to branch-2.4: Resolved