[HDDS-6441] Ozone metadata does not align with underlying blocks when there are many incomplete uploads happens

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Component/s: Ozone Datanode
• Assignee: Ethan Rose
• Reporter: Shawn
• Created: 12/Mar/22 05:23
• Updated: 29/Nov/23 05:04
• Resolved: 23/Mar/23 18:38

Description

Ozone metadata does not align with underlying blocks when there are many incomplete uploads happens. I have a cluster which has a very few objects. But the datanode usage tells me I almost run out of space.

????
Usage info for datanode with UUID f50108f1-d8bf-44e3-abed-6e77c91f994d:
Capacity : 8802545958912B
SCMUsed : 8802128257024B (99.99525%)
Remaining : 74715136B (0.00085%)

Usage info for datanode with UUID 2bdb3198-b71f-4153-9663-e3b349c6f82a:
Capacity : 8802545958912B
SCMUsed : 8802133102592B (99.99531%)
Remaining : 76824576B (0.00087%)

Usage info for datanode with UUID d5644a36-b967-44a6-a736-4bd2013c2b86:
Capacity : 8793955991552B
SCMUsed : 8793311227904B (99.99267%)
Remaining : 291676160B (0.00332%)
...
 

Also I see there are lots of errors in logs, complaining out of disk space and also report missing .container files as below:

????
2022-03-10 03:56:02 ERROR Thread-6 ContainerReader:159 - Missing .container file for ContainerID: 15221
 

Issue Links

• is related to: HDDS-9095 Cleanup expired incomplete MPUs
• split to: HDDS-6449 Failed container delete can leave artifacts on disk
• split to: HDDS-6721 Ozone key deletion often dose not delete the real data blocks in datanode