[HDDS-2794] Failed to close QUASI_CLOSED container

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Assignee: Unassigned
• Reporter: Sammi Chen
• Created: 25/Dec/19 03:30
• Updated: 09/Jun/20 16:34
• Resolved: 09/Jun/20 16:34

Description

2019-12-24 20:19:55,154 INFO org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Process replica:ContainerReplica{containerID=#283, datanodeDetails=ed90869c-317e-4303-8922-9fa83a3983cb

{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}

, placeOfBirth=ed90869c-317e-4303-8922-9fa83a3983cb, sequenceId=2342, state=QUASI_CLOSED}

2019-12-24 20:20:02,258 INFO org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Process replica:ContainerReplica{containerID=#283, datanodeDetails=1da74a1d-f64d-4ad4-b04c-85f26687e683

{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}

, placeOfBirth=1da74a1d-f64d-4ad4-b04c-85f26687e683, sequenceId=2209, state=UNHEALTHY}

2019-12-24 20:20:03,167 INFO org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Process replica:ContainerReplica{containerID=#283, datanodeDetails=b65b0b6c-b0bb-429f-a23d-467c72d4b85c

{ip: 10.120.139.111, host: host111, networkLocation: /rack1, certSerialId: null}

, placeOfBirth=b65b0b6c-b0bb-429f-a23d-467c72d4b85c, sequenceId=2209, state=UNHEALTHY}

2019-12-24 20:20:03,168 INFO org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler: Close container Event triggered for container : #283
2019-12-24 20:20:03,169 WARN org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler: Cannot close container #283, which is in QUASI_CLOSED state.

ozone scmcli container list -s=283

{ "state" : "QUASI_CLOSED", "replicationFactor" : "THREE", "replicationType" : "RATIS", "usedBytes" : 872715244, "numberOfKeys" : 9, "lastUsed" : 14385015083, "stateEnterTime" : 14313955037, "owner" : "d0e31665-ba27-45ad-b576-67cd1bccc50b", "containerID" : 283, "deleteTransactionId" : 0, "sequenceId" : 0, "open" : false }

ozone scmcli container info 283
Loaded properties from hadoop-metrics2.properties
Scheduled Metric snapshot period at 10 second(s).
XceiverClientMetrics metrics system started
Container id: 283
Container State: CLOSED
Container Path: /data5/hdds/df508c61-3ae7-413f-ab9d-e00d9125de70/current/containerDir0/283/metadata
Container Metadata:
Datanodes: [host172]