[HDDS-8018] Datanode kept getting container deletion command for non-empty container

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: SCM
• Assignee: Unassigned
• Reporter: George Huang
• Created: 23/Feb/23 04:51
• Updated: 13/Mar/23 16:01
• Resolved: 13/Mar/23 16:01

Description

9:46:21.461 AM	ERROR	KeyValueHandler	Received container deletion command for container 50365 but the container is not empty.
9:46:21.461 AM	ERROR	KeyValueHandler	Received container deletion command for container 49154 but the container is not empty.
9:46:21.462 AM	ERROR	DeleteContainerCommandHandler	Exception occurred while deleting the container. org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Non-force deletion of non-empty container is not allowed. at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteInternal(KeyValueHandler.java:1237) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteContainer(KeyValueHandler.java:1149) at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.deleteContainer(ContainerController.java:182) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.handleInternal(DeleteContainerCommandHandler.java:108) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.lambda$handle$0(DeleteContainerCommandHandler.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
9:46:21.462 AM	ERROR	DeleteContainerCommandHandler	Exception occurred while deleting the container. org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Non-force deletion of non-empty container is not allowed. at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteInternal(KeyValueHandler.java:1237) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteContainer(KeyValueHandler.java:1149) at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.deleteContainer(ContainerController.java:182) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.handleInternal(DeleteContainerCommandHandler.java:108) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.lambda$handle$0(DeleteContainerCommandHandler.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
10:21:21.474 AM	ERROR	KeyValueHandler	Received container deletion command for container 49154 but the container is not empty.
10:21:21.474 AM	ERROR	KeyValueHandler	Received container deletion command for container 50365 but the container is not empty.
10:21:21.474 AM	ERROR	DeleteContainerCommandHandler	Exception occurred while deleting the container. org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Non-force deletion of non-empty container is not allowed. at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteInternal(KeyValueHandler.java:1237) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteContainer(KeyValueHandler.java:1149) at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.deleteContainer(ContainerController.java:182) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.handleInternal(DeleteContainerCommandHandler.java:108) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.lambda$handle$0(DeleteContainerCommandHandler.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
10:21:21.474 AM	ERROR	DeleteContainerCommandHandler	Exception occurred while deleting the container. org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Non-force deletion of non-empty container is not allowed. at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteInternal(KeyValueHandler.java:1237) at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.deleteContainer(KeyValueHandler.java:1149) at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.deleteContainer(ContainerController.java:182) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.handleInternal(DeleteContainerCommandHandler.java:108) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.DeleteContainerCommandHandler.lambda$handle$0(DeleteContainerCommandHandler.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Issue Links

• duplicates: HDDS-8141 Exception "Non-force deletion of non-empty container is not allowed" in datanode logs