[HDFS-14513] FSImage which is saving should be clean while NameNode shutdown

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Xiaoqiao He
• Reporter: Xiaoqiao He
• Created: 26/May/19 09:29
• Updated: 06/Aug/19 16:51
• Resolved: 11/Jun/19 18:49

Description

Checkpointer/FSImageSaver is regular tasks and dump NameNode meta to disk, at most per hour by default. If it receive some command (e.g. transition to active in HA mode) it will cancel checkpoint and delete tmp files using FSImage#deleteCancelledCheckpoint. However if NameNode shutdown when checkpoint, the tmp files will not be cleaned anymore.
Consider there are 500m inodes+blocks, it could cost 5~10min to finish once checkpoint, if we shutdown NameNode during checkpointing, fsimage checkpoint file will never be cleaned, after long time, there could be many useless checkpoint files. So I propose that we should add hook to clean that when shutdown.

Issue Links

• relates to: HDFS-14697 Backport HDFS-14513 to branch-2