[HBASE-27251] Rolling back from 2.5.0-SNAPSHOT to 2.4.13 fails due to `File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo`

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.0
• Fix Version/s: 2.4.14
• Component/s: master

Release Note

Before upgrading HBase cluster to 2.5.0, it is strongly recommended to upgrade the cluster to HBase-2.4.14 or later first. This is to guarantee a smooth rollback in case. Otherwise, it may run into HBASE-27251.

• Assignee: Huaxiang Sun
• Reporter: Nick Dimiduk
• Created: 27/Jul/22 17:52
• Updated: 03/Aug/22 04:49
• Resolved: 01/Aug/22 23:51

Description

I was doing some perf testing with builds of 2.5.0. I rolled back to 2.4.13 and the master won't start. Stack trace ends in,

java.io.FileNotFoundException: File does not exist: /hbase/MasterData/data/master/store/.initialized/.regioninfo                                                                                          
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)                                                                                                                            at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
        at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)                                                                                         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2089)                                                                                                  
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:762)                                                                                         
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:458)                                        
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)                                                      at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572)                                                                                           
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)                               
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)                                                                                                                                           
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)                                                                                                                                     
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) 
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)                                                                                                               
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439)                                                                                                                                   
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)                                                                                                           
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

When I examine the on-disk file system, I see,

nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/
Found 3 items
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.initialized
drwxr-xr-x   - nonroot supergroup          0 2022-07-19 17:37 /hbase/MasterData/data/master/store/.tabledesc
drwxr-xr-x   - nonroot supergroup          0 2022-07-27 16:25 /hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682
nonroot@namenode-0:~$ hdfs dfs -ls /hbase/MasterData/data/master/store/.initialized/
nonroot@namenode-0:~$

Issue Links

• relates to: HBASE-26640 Reimplement master local region initialization to better work with SFT
• links to: GitHub Pull Request #4663
• links to: GitHub Pull Request #4665