[HDDS-9544] Incorrect pipeline ID and state for closed container.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Aryan Gupta
• Reporter: Pratyush Bhatt
• Created: 26/Oct/23 07:14
• Updated: 21/Nov/23 10:34
• Resolved: 21/Nov/23 10:32

Description

Steps:
1. Put key in a ozone bucket.
2. Stop one of the 3 replica DN.
3. Let the re-replicaiton happen, the container will be in CLOSED state post this.
4. do a container info on this closed container.

Observed behavior: 
The pipeline ID keeps on changing on every request.

[root@ozn-decom75-1 data184295]# ozone admin container info 2007
Container id: 2007
Pipeline id: fa883a43-65d8-4f37-b3ba-d4a3ae1158b1
Container State: CLOSED
Datanodes: [93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz]
Replicas: [State: CLOSED; ReplicaIndex: 0; Origin: 0e65f84a-2920-46dc-a61b-3199aaa1293c; Location: 93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz]
[root@ozn-decom75-1 data184295]#
[root@ozn-decom75-1 data184295]#
[root@ozn-decom75-1 data184295]# ozone admin container info 2007
Container id: 2007
Pipeline id: babe680e-53be-4aa5-a43f-8efd1d2e2699
Container State: CLOSED
Datanodes: [93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz]
Replicas: [State: CLOSED; ReplicaIndex: 0; Origin: 0e65f84a-2920-46dc-a61b-3199aaa1293c; Location: 93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz]
[root@ozn-decom75-1 data184295]#
[root@ozn-decom75-1 data184295]#
[root@ozn-decom75-1 data184295]# ozone admin container info 2007
Container id: 2007
Pipeline id: 60bce535-4197-4de5-9b6a-b5a3e3d1580e
Container State: CLOSED
Datanodes: [93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz]
Replicas: [State: CLOSED; ReplicaIndex: 0; Origin: 0e65f84a-2920-46dc-a61b-3199aaa1293c; Location: 93650eaf-a971-4d97-ac3c-332acc9e9904/ozn-decom75-9.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6/ozn-decom75-6.ozn-decom75.xyz,
State: CLOSED; ReplicaIndex: 0; Origin: 351c6c1a-e0f1-4567-b35f-cd70bb4643a6; Location: 21c818e9-f198-42dd-bb88-5de0293a80c8/ozn-decom75-5.ozn-decom75.xyz] 

Can see above that on 3 requests, there are 3 different pipeline IDs. 

sumitagrawal  helped in debugging(Thanks!), 
When we check the pipeline ID for container 2007 using:

ozone debug ldb --db scm.db scan --with-keys --column_family=containers

it shows below:

{
  "state": "CLOSED",
  "stateEnterTime": {
    "seconds": 1698211757,
    "nanos": 92000000
  },
  "pipelineID": {
    "id": "729e1518-467e-4eb5-b8ef-f4a61dc03920"
  },
  "replicationConfig": {
    "replicationFactor": "THREE"
  },
  "clock": {
    "zone": {
      "totalSeconds": 0
    }
  },
  "usedBytes": 0,
  "numberOfKeys": 0,
  "lastUsed": {
    "seconds": 1698303048,
    "nanos": 731000000
  },
  "owner": "om146",
  "containerID": {
    "id": 2007
  },
  "deleteTransactionId": 61,
  "sequenceId": 0
},

here the pipeline ID is "729e1518-467e-4eb5-b8ef-f4a61dc03920"
Looks like it randomly generates the pipelineID on every request, which should not be the case.

Issue Links

• links to: GitHub Pull Request #5541