[HDDS-9005] Container scanner continues to scan deleted container

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Labels: pull-request-available
• Assignee: Saketa Chalamchala
• Reporter: George Huang
• Created: 13/Jul/23 00:26
• Updated: 03/Jan/24 16:08
• Resolved: 03/Jan/24 16:08

Description

The following was observed in the log of a datanode running the container scanner:

2023-07-12 06:04:39,049 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,061 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,062 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,063 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,064 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,065 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,066 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] ERROR org.apache.hadoop.ozone.container.common.helpers.ContainerUtils: Chunks dir /data/2/hadoop-ozone/datanode/data/hdds/CID-c4cbea3d-ac0b-4956-a8cc-6fd5f3a55ec4/current/containerDir2/1104/chunks does not exist
2023-07-12 06:04:39,066 [ContainerDataScanner(/data/2/hadoop-ozone/datanode/data/hdds)] WARN org.apache.hadoop.ozone.container.ozoneimpl.ContainerController: Container 1104 not found, may be deleted, skip update DataScanTimestamp

ozone admin container info showed that the container replica was no longer present on the datanode that logged these messages. The replica was likely deleted due to over-replication while the scanner was running.

Issue Links

• relates to: HDDS-7300 Race condition between full data scan and block deletion
• requires: HDDS-8770 Cleanup of failed container delete may remove datanode RocksDB entries of active container
• links to: GitHub Pull Request #5485