[HDDS-6434] EC: Overwriting an EC key with a Ratis key fails

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: EC-Branch
• Labels: pull-request-available
• Assignee: Uma Maheswara Rao G
• Reporter: Stephen O'Donnell
• Created: 09/Mar/22 22:21
• Updated: 08/Apr/22 05:53
• Resolved: 08/Apr/22 05:53

Description

Creating a key with Replication Config EC, and then overwiting the key with a Ratis key fails with the error:

bash-4.2$ ozone sh key put -t EC -r rs-3-2-1024k /vol1/bucket1/key11 /etc/hosts
bash-4.2$ ozone sh key put -t RATIS -r THREE /vol1/bucket1/key11 /etc/hosts
Required instance of class org.apache.hadoop.hdds.client.ECReplicationConfig but object.getClass() is class org.apache.hadoop.hdds.client.RatisReplicationConfig

If you repeat this test with a RATIS/ONE key and overwrite with RATIS/THREE, the key info continues to have a replication config of ONE, but the first version goes into a ONE container. Then the next version goes into a THREE container.

bash-4.2$ ozone sh key put -t RATIS -r ONE /vol1/bucket1/key40 3mb
bash-4.2$ ozone sh key put -t RATIS -r THREE /vol1/bucket1/key40 3mb

bash-4.2$ ozone sh key info /vol1/bucket1/key40 
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "key40",
  "dataSize" : 3145728,
  "creationTime" : "2022-03-09T21:53:29.492Z",
  "modificationTime" : "2022-03-09T21:53:39.109Z",
  "replicationConfig" : {
    "replicationFactor" : "ONE",
    "requiredNodes" : 1,
    "replicationType" : "RATIS"
  },
  "ozoneKeyLocations" : [ {
    "containerID" : 20,
    "localID" : 109611004723200035,
    "length" : 3145728,
    "offset" : 0,
    "keyOffset" : 0
  } ],
  "metadata" : { }
}

Current container is 20:

bash-4.2$ ozone admin container info 20       
Container id: 20
Pipeline id: 35b541ec-b50f-49ac-ae93-c8bdac3dd659
Container State: OPEN
Datanodes: [7465dd7d-6ee7-4c5b-b9f0-5d63b3bdbf09/ozone_datanode_4.ozone_default,
71c5c10f-6d38-428e-85aa-7202028e3c1c/ozone_datanode_2.ozone_default,
25c43ccd-b027-43f0-b3cc-88e415390e86/ozone_datanode_5.ozone_default]
Replicas: [State: OPEN; ReplicaIndex: 0; Origin: 25c43ccd-b027-43f0-b3cc-88e415390e86; Location: 25c43ccd-b027-43f0-b3cc-88e415390e86/ozone_datanode_5.ozone_default,
State: OPEN; ReplicaIndex: 0; Origin: 71c5c10f-6d38-428e-85aa-7202028e3c1c; Location: 71c5c10f-6d38-428e-85aa-7202028e3c1c/ozone_datanode_2.ozone_default,
State: OPEN; ReplicaIndex: 0; Origin: 7465dd7d-6ee7-4c5b-b9f0-5d63b3bdbf09; Location: 7465dd7d-6ee7-4c5b-b9f0-5d63b3bdbf09/ozone_datanode_4.ozone_default]

Write an EC key - note container changes to 22, but repConfig is still RATIS/ONE in the key info, and container 22 is RATIS/ONE

ozone sh key put -t EC -r rs-3-2-1024k /vol1/bucket1/key40 3mb  

bash-4.2$ ozone sh key info /vol1/bucket1/key40 
{
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "key40",
  "dataSize" : 3145728,
  "creationTime" : "2022-03-09T21:53:29.492Z",
  "modificationTime" : "2022-03-09T21:57:10.909Z",
  "replicationConfig" : {
    "replicationFactor" : "ONE",
    "requiredNodes" : 1,
    "replicationType" : "RATIS"
  },
  "ozoneKeyLocations" : [ {
    "containerID" : 22,
    "localID" : 109611004723200037,
    "length" : 3145728,
    "offset" : 0,
    "keyOffset" : 0
  } ],
  "metadata" : { }
  }

Even more strangely, there is a container 21 created, which is EC and has 3MB written to the first replica, and hence not correctly stripped across the blocks. Its also not stored in the container:

bash-4.2$ ozone admin container info 21 --json
  ...
    "placeOfBirth" : "25c43ccd-b027-43f0-b3cc-88e415390e86",
    "sequenceId" : 0,
    "keyCount" : 0,
    "bytesUsed" : 3145728,
    "replicaIndex" : 1

Its like it writes a RATIS/ONE and also writes the data to an EC container too

Issue Links

• links to: GitHub Pull Request #3274