[HDFS-11304] Namenode fails to start, even edit log available in the journal node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.8.0, 2.7.1
• Component/s: hdfs, journal-node
• Labels: patch

Environment

HDP 2.4.2.0-258

• Assignee: Karthik Palanisamy
• Reporter: Karthik Palanisamy
• Created: 09/Jan/17 17:29
• Updated: 02/Oct/19 17:14
• Resolved: 20/Mar/19 19:27

Description

JN => JournalNode
NN => Namenode local directory (dfs.namenode.name.dir)
Y/N => Is edit file/log present?
Ex : edits_0000000000001627921-0000000000001627961

Scenario:

JN 1	JN 2	JN 3	NN local	Is NN started?
N	N	Y	N	Started
Y	N	N	N	Started
N	Y	N	N	Failed
N	Y	N	Y	Started
Y	Y	N	N	Started

Note: Namenode and JN2 installed on the same machine

Trace :

ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.io.IOException: There appears to be a gap in the edit log. We expected txid 1627921, but got txid 1627962.
at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)