[HDFS-15421] IBR leak causes standby NN to be stuck in safe mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.2, 2.10.1
• Component/s: namenode
• Labels: release-blocker
• Target Version/s: 3.3.0, 3.2.2, 2.10.1
• Assignee: Akira Ajisaka
• Reporter: Kihwal Lee
• Created: 19/Jun/20 17:31
• Updated: 04/Apr/24 22:39
• Resolved: 28/Jun/20 11:25

Description

After HDFS-14941, update of the global gen stamp is delayed in certain situations.  This makes the last set of incremental block reports from append "from future", which causes it to be simply re-queued to the pending DN message queue, rather than processed to complete the block.  The last set of IBRs will leak and never cleaned until it transitions to active.  The size of pendingDNMessages constantly grows until then.

If a leak happens while in a startup safe mode, the namenode will never be able to come out of safe mode on its own.

Issue Links

• is broken by: HDFS-14941 Potential editlog race condition can cause corrupted file
• relates to: HDFS-17453 IncrementalBlockReport can have race condition with Edit Log Tailer