[HDDS-3009] Create Container failed using a disk which is full

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: teragentest
• Target Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Bharat Viswanadham
• Created: 13/Feb/20 21:49
• Updated: 21/Dec/22 08:58

Description

/dev/nvme2n1    985G  935G     0 100% /ozone-data

/dev/nvme4n1    985G  453G  482G  49% /ozone-data1

 

The container path used is full, but in creating container we have used the disk which is filled up 100% and failing during creating container path creation.

 

hdds.datanode.dir is configured with /ozone-data, /ozone

 

 

2020-02-13 10:58:01,097 ERROR org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil: Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
2020-02-13 10:58:01,097 INFO org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation: CreateContainer , Trace ID:  , Message: Container creation failed. Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container creation failed. Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:177)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:244)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:164)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:412)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:248)
        at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:162)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:396)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:406)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$2(ContainerStateMachine.java:441)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to create directory for metadata storage. Path: /ozone-data/hdds/cf793a84-8529-4897-8bf9-f18ec79e8ad6/current/containerDir1/536/metadata
        at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.createContainerMetaData(KeyValueContainerUtil.java:73)
        at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:142)
        ... 12 more