[HDDS-4966] Disk space leak after deleting Ozone files

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Wei-Chiu Chuang
• Created: 11/Mar/21 02:29
• Updated: 02/Aug/23 10:44
• Resolved: 02/Aug/23 10:44

Description

Sometimes deleting Ozone files does not remove containers and block files at all. Even removing the bucket doesn't reclaim the space. I even forcefully closed all the containers.

Due to HDDS-2534, there is no way to delete the dangling containers manually.

What makes it even more perplexing is that some containers are no longer in SCM's record (ozone admin container list couldn't find the container) but the container replica are still (including block files and metadata) on the DNs.

Looking at OM's rocksdb instance, keyTable is empty, bucketTable is empty. prefixKeyTable
empty. But openKeyTable is not.

The application that generated the data (teragen) was killed. I don't think the clients are still connected and the openKeyTable is not pruned after a day.

Maybe we shouldn't prune openKeyTable when deleting the corresponding volume/bucket?

Unfortunately, it happens from time to time but i can't reliably reproduce.

Issue Links

• duplicates: HDDS-4961 Disk space leak after deleting Ozone files