[HDFS-16774] Improve async delete replica on datanode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: datanode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Haiyang Hu
• Reporter: Haiyang Hu
• Created: 17/Sep/22 02:50
• Updated: 28/Jan/24 08:04
• Resolved: 11/Oct/22 08:25

Description

In our online cluster, a large number of ReplicaNotFoundExceptions occur when client reads the data.
After tracing the root cause, it is caused by the asynchronous deletion of the replica operation has many stacked pending deletion caused ReplicationNotFoundException.
Current the asynchronous delete of the replica operation process is as follows：
1.remove the replica from the ReplicaMap
2.delete the replica file on the disk [blocked in threadpool]
3.notifying namenode through IBR [blocked in threadpool]

In order to avoid similar problems as much as possible, consider optimizing the execution flow：
The deleting replica from ReplicaMap, deleting replica from disk and notifying namenode through IBR are processed in the same asynchronous thread.

Issue Links

• links to: GitHub Pull Request #4903