[HDFS-15568] namenode start failed to start when dfs.namenode.snapshot.max.limit set

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: snapshots
• Labels: pull-request-available
• Assignee: Shashikant Banerjee
• Reporter: Nilotpal Nandi
• Created: 10/Sep/20 08:15
• Updated: 17/Sep/20 09:20
• Resolved: 17/Sep/20 09:20

Description

11:35:05.872 AM	ERROR	NameNode	
Failed to start namenode.
org.apache.hadoop.hdfs.protocol.SnapshotException: Failed to add snapshot: there are already 20 snapshot(s) and the max snapshot limit is 20
	at org.apache.hadoop.hdfs.server.namenode.snapshot.DirectorySnapshottableFeature.addSnapshot(DirectorySnapshottableFeature.java:181)
	at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSnapshot(INodeDirectory.java:285)
	at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotManager.createSnapshot(SnapshotManager.java:447)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:802)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:287)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1164)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:755)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:646)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:717)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:960)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:933)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1737)

Steps to reproduce:

----------------------

directory level snapshot limit set - 100
Created 100 snapshots
deleted all 100 snapshots (in-oder)
No snapshot exist

Then, directory level snapshot limit set - 20
HDFS restart
Namenode start failed.

Issue Links

• links to: GitHub Pull Request #2296