[HDDS-8129] ContainerStateMachine allows two different tasks with the same container id running in parallel

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 09/Mar/23 22:23
• Updated: 02/Aug/23 18:16
• Resolved: 10/Mar/23 22:11

Description

In the ContainerStateMachine.submitTask(..) method, we can have the following case

1. The applyTransaction thread enters submitTask(..). It gets a queue Q for a container C from containerTaskQueues, where Q is empty at that time.
2. Right before applyTransaction submits a task T1, the executor has completed running a previous task T2 for container C. Since Q is empty, the executor removes Q from containerTaskQueues.
3. The applyTransaction thread submits task T1 to Q.  It does not aware that Q is no longer in containerTaskQueues.
4. Now, the applyTransaction thread submits a new task T3 for container C. Since there is no queue for container C in containerTaskQueues, it creates a new queue Q2 and submits T3 to Q2.

As a result, T1 and T3 for container C can run in parallel since there are two queues.

Issue Links

• causes: HDDS-8141 Exception "Non-force deletion of non-empty container is not allowed" in datanode logs
• is related to: HDDS-8141 Exception "Non-force deletion of non-empty container is not allowed" in datanode logs
• is related to: HDDS-4352 Container info is not correct when there is QUASI_CLOSED replica
• relates to: HDDS-7701 Data loss by deleting containers having non-deleted keys
• relates to: HDDS-5359 Incorrect BLOCKCOUNT and BYTESUSED in container DB
• links to: GitHub Pull Request #4370