[HBASE-26267] Master initialization fails if Master Region WAL dir is missing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.6
• Fix Version/s: 2.5.0, 3.0.0-alpha-2, 2.4.9
• Component/s: master
• Hadoop Flags: Reviewed
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 08/Sep/21 20:46
• Updated: 17/Nov/21 18:41
• Resolved: 16/Nov/21 21:42

Description

From a recent branch-2.4 build:

2021-09-07 19:31:19,666 ERROR [master/localhost:16000:becomeActiveMaster] master.HMaster(159): ***** ABORTING master localhost,16000,1631057476442: Unhandled exception. Starting shutdown. *****
java.io.FileNotFoundException: File hdfs://localhost:8020/hbase-2.4-wals/MasterData/WALs does not exist.
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1059)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
        at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1119)
        at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1116)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1126)
        at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:226)
        at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:303)
        at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:839)
        at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2189)
        at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:512)
        at java.lang.Thread.run(Thread.java:748)

If the WAL directory is missing but the Master Region already exists, we will try to list the contents of the Master Region's WAL directory which may or may not exist. If we simply check to make sure the directory exists and then the rest of the initialization code works as expected.

Issue Links

• links to: GitHub Pull Request #3679