[HDDS-8865] Ozone asynchronous deletion performance optimization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: ChenXi
• Reporter: ChenXi
• Created: 16/Jun/23 06:10
• Updated: 03/Oct/24 16:15

Description

Background:
We have a Cluster which will write many small key. The user will merge these small keys into one big key, so these small keys will not be stored for a long time. So we have almost the same QPS of delete requests as writes.
The current key deletion performance cannot meet the requirements, the disk is using less and less available capacity, but in fact, only a small portion of the user's valid data

Issue Links

• is related to: HDDS-11506 Improvements for large scale deletion

Sub-Tasks

• Support CommandHandler Metrics: Resolved
• Make DN DeleteBlocksCommandHandler wait for the lock can timeout.: Resolved
• Reduce duplicate delete transactions write to the DB: Open
• Add more BlockDeletingServiceMetrics to Monitor the Datanode deleting process.: Resolved
• Add state management of SCM's DeleteBlocksCommand to avoid sending duplicate delete transactions to the DN: Resolved
• Limit SCMBlockDeletingService to sending delete transactions to Datanode whose queue is full: Resolved
• Improve SCM deletion efficiency: Resolved