[HDFS-17218] NameNode should process time out excess redundancy blocks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: namenode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Haiyang Hu
• Reporter: Haiyang Hu
• Created: 08/Oct/23 04:00
• Updated: 16/Oct/24 18:29
• Resolved: 04/Dec/23 13:17

Description

Currently found that DN will lose all pending DNA_INVALIDATE blocks if it restarts.

Root case
Current DN enables asynchronously deletion, it have many pending deletion blocks in memory.
when DN restarts, these cached blocks may be lost. it causes some blocks in the excess map in the namenode to be leaked and this will result in many blocks having more replicas then expected.

solution
NameNode add logic to handle excess redundant block timeouts to resolve current issue.
If NN determines that the excess redundancy block in DN has timed out and re-adds it to Invalidates.

Issue Links

• links to: GitHub Pull Request #6176
• links to: GitHub Pull Request #6233