[HDDS-7328] Improve Deletion of FSO Paths

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Labels: pull-request-available
• Assignee: Xu Shao Hong
• Reporter: Xu Shao Hong
• Created: 14/Oct/22 02:36
• Updated: 16/Feb/23 14:38
• Resolved: 27/Oct/22 12:03

Description

Currently, the deleting speed of paths in the FSO bucket is too slow.

Every time the DirectoryDeletingService executes(every 1m by default),  it chooses only one path from DeletedDirectroyTable and tries adding its sub-files and sub-dirs into the PurgeDirectories Request. The real deletion happens in OMKeyDeleteResponseWithFSO, which moved all sub-dirs into DeletedDirectroyTable.  Then these sub-dirs will be chosen by the future execution of DirectoryDeletingService.

In the real production environment, such deletion speed is not applicable. There could be lots of dirs deleted without sub-dirs or sub-files. The dir deletion speed would lag behind the dir creation speed.

I propose to optimize the logic of the DirectoryDeletingService, we could consume all the quota (ozone.path.deleting.limit.per.task) to delete paths as much as possible. The good news is the PB is already designed to do so, we could add more PurgePathRequest in one PurgeDirectoriesRequest.

-----------------------------------

This should be compatible with HDDS-5048.

This aims to improve the efficiency of the deletion if not enough sub-files or sub-dirs are chosen.

Issue Links

• causes: HDDS-7447 Leak of TableIterator in DirectoryDeletingService.
• is related to: HDDS-5048 [FSO] Perf optimization - Recursively visit sub-path components and do cleanup
• is related to: HDDS-7541 FSO recursive delete directory with hierarchy takes much time for cleanup
• is related to: HDDS-7147 DirectoryDeletingService interval unit mismatch
• is related to: HDDS-7592 Recursive delete of directory multiple time impacts some directory not removed
• links to: GitHub Pull Request #3844