[HBASE-16056] Procedure v2 - fix master crash for FileNotFound

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.2.1, 1.1.5, 2.0.0
• Fix Version/s: 1.3.0, 1.2.2, 1.1.6, 2.0.0
• Component/s: proc-v2
• Assignee: Matteo Bertozzi
• Reporter: Matteo Bertozzi
• Created: 17/Jun/16 07:39
• Updated: 17/Jun/22 18:16
• Resolved: 17/Jun/16 20:13

Description

syuanjiang and tedyu reported a backup master not able to start with FileNotFound during proc-v2 lease recovery. (another restart should have solved the problem)

FileNotFoundException: File does not exist: /hbase/MasterProcWALs/state-000001.log
namenode.INodeFile.valueOf(INodeFile.java:61) at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) 

this may happen when the other master is still active (e.g. GC) and tries to remove files while the other master tries to become active. This operation is retryable so the code should able to handle that.