[HDFS-17003] Erasure Coding: invalidate wrong block after reporting bad blocks from datanode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.6
• Fix Version/s: 3.4.0, 3.3.6
• Component/s: namenode
• Labels: pull-request-available
• Target Version/s: 3.3.6
• Hadoop Flags: Reviewed
• Assignee: farmmamba
• Reporter: farmmamba
• Created: 08/May/23 12:42
• Updated: 28/Jan/24 02:19
• Resolved: 08/Jun/23 10:08

Description

After receiving reportBadBlocks RPC from datanode, NameNode compute wrong block to invalidate. It is a dangerous behaviour and may cause data loss. Some logs in our production as below:

 

NameNode log:

2023-05-08 21:23:49,112 INFO org.apache.hadoop.hdfs.StateChange: *DIR* reportBadBlocks for block: BP-932824627-xxxx-1680179358678:blk_-9223372036848404320_1471186 on datanode: datanode1:50010

2023-05-08 21:23:49,183 INFO org.apache.hadoop.hdfs.StateChange: *DIR* reportBadBlocks for block: BP-932824627-xxxx-1680179358678:blk_-9223372036848404319_1471186 on datanode: datanode2:50010

datanode1 log:

2023-05-08 21:23:49,088 WARN org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad BP-932824627-xxxx-1680179358678:blk_-9223372036848404320_1471186 on /data7/hadoop/hdfs/datanode

2023-05-08 21:24:00,509 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed to delete replica blk_-9223372036848404319_1471186: ReplicaInfo not found.

 

This phenomenon can be reproduced.

Issue Links

• links to: GitHub Pull Request #5643