[HDDS-11510] All deletion services should support multiple threads

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM, SCM
• Assignee: Aryan Gupta
• Reporter: Ethan Rose
• Created: 30/Sep/24 20:50
• Updated: 4 days ago 10:32

Description

When a large amount of data is deleted, we may see slow progress to clear it out due to the large backlog of pending deletes, even if all deletion services are running for their full intervals and collecting their maximum number of entries. In order to speed up deletion further, more threads will need to be configured. However, the deletion algorithms in OM and SCM currently only support single threaded implementations. The algorithms for these jobs need to be updates to partition the pending delete key space to support any number of deletion threads working over it.

This Jira is relevant for the following components and these may be done in separate subtasks:

• OM open key cleanup
• OM key delete
• OM directory delete
• SCM block delete
  Datanode block deletion already supports a configurable thread pool that divides work by container.
   

Issue Links

• is required by: HDDS-11511 OM deletion services should have consistent metrics
• is required by: HDDS-11513 All deletion configurations should be configurable without restart
• Parent Feature: HDDS-11605 Directory deletion service should support multiple threads