[HDDS-6279] EC: Calculate EC replication correctly when updating bucket usage

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: EC-Branch
• Labels: pull-request-available
• Assignee: Stephen O'Donnell
• Reporter: Kaijie Chen
• Created: 08/Feb/22 08:39
• Updated: 25/Feb/22 20:32
• Resolved: 25/Feb/22 20:32

Description

The usedBytes of EC keys is incorrect.

For an EC key with d+p blocks, usedBytes is   dataSize * (d + p)   now.

It should be   (dataSize + padding) * (d + p) / d   instead.

 

For example, writing a 3:2 EC key with 3,000,000 bytes, usedBytes should be 5,000,000 instead of 15,000,000.

$ bin/ozone sh bucket create /vol1/bucket1 -t EC -r rs-3-2-1024k
$ bin/ozone sh key put /vol1/bucket1/key1 ~/keys/key1 
$ bin/ozone sh key list /vol1/bucket1
[ {
  "volumeName" : "vol1",
  "bucketName" : "bucket1",
  "name" : "key1",
  "dataSize" : 3000000,
  "creationTime" : "2022-02-08T08:35:30.945Z",
  "modificationTime" : "2022-02-08T08:35:32.029Z",
  "replicationConfig" : {
    "data" : 3,
    "parity" : 2,
    "ecChunkSize" : 1048576,
    "codec" : "RS",
    "requiredNodes" : 5,
    "replicationType" : "EC"
  },
  "replicationType" : "EC",
  "replicationFactor" : 5
} ]
$ bin/ozone sh bucket list /vol1
[ {
  "metadata" : { },
  "volumeName" : "vol1",
  "name" : "bucket1",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 15000000,
  "usedNamespace" : 1,
  "creationTime" : "2022-02-08T08:12:22.897Z",
  "modificationTime" : "2022-02-08T08:12:22.897Z",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "bucketLayout" : "OBJECT_STORE",
  "owner" : "ckj",
  "link" : false,
  "replicationConfig" : {
    "data" : 3,
    "parity" : 2,
    "ecChunkSize" : 1048576,
    "codec" : "RS",
    "replicationType" : "EC",
    "requiredNodes" : 5
  }
} ]

 

Issue Links

• links to: GitHub Pull Request #3136