[HDDS-541] Ozone Quota support.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: Triaged
• Assignee: mingchao zhao
• Reporter: Namit Maheshwari
• Created: 23/Sep/18 02:18
• Updated: 21/Jan/24 15:48
• Resolved: 01/Mar/23 18:00

Description

Create a volume with just 1 MB as quota

[root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh volume create --quota=1MB --user=root /hive
2018-09-23 02:10:11,283 [main] INFO - Creating Volume: hive, with root as owner and quota set to 1048576 bytes.

Now create a bucket and put a big key greater than 1MB in the bucket

[root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh bucket create /hive/bucket1
2018-09-23 02:10:38,003 [main] INFO - Creating Bucket: hive/bucket1, with Versioning false and Storage Type set to DISK
[root@ctr-e138-1518143905142-481027-01-000002 bin]# ls -l ../../ozone-0.3.0-SNAPSHOT.tar.gz
-rw-r--r-- 1 root root 165903437 Sep 21 13:16 ../../ozone-0.3.0-SNAPSHOT.tar.gz
[root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key put /hive/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
volume/bucket/key name required in putKey
[root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key put /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
[root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key info /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz
{
"version" : 0,
"md5hash" : null,
"createdOn" : "Sun, 23 Sep 2018 02:13:02 GMT",
"modifiedOn" : "Sun, 23 Sep 2018 02:13:08 GMT",
"size" : 165903437,
"keyName" : "ozone-0.3.0-SNAPSHOT.tar.gz",
"keyLocations" : [ {
"containerID" : 2,
"localID" : 100772661343420416,
"length" : 134217728,
"offset" : 0
}, {
"containerID" : 3,
"localID" : 100772661661007873,
"length" : 31685709,
"offset" : 0
} ]
}

It was able to put a 165 MB file on a volume with just 1MB quota.

 

Currently Ozone haven't support Quota, So I think this should be a new feature .
The design document can be referred to the attachment. (design google docs)

Issue Links

• is duplicated by: HDDS-3648 Volume quota is not enforced
• relates to: HDDS-4308 Fix issue with quota update

Sub-Tasks

• Ozone sh volume client support quota option.: Resolved
• Volume space: add quotaUsageInBytes and update it when create and delete key: Resolved
• Add protobuf field to support quota.: Closed
• Volume space: check quotaUsageInBytes when write key: Resolved
• Ozone sh bucket client support quota option.: Resolved
• Bucket space: Supports clearing spaceQuota: Resolved
• Volume space: Supports clearing spaceQuota: Resolved
• Bucket space: add usedBytes and update it when create and delete key: Resolved
• Bucket space: check quotaUsageInBytes when write key: Resolved
• Space Quota: bucket quota not exceeded, volume needn’t check.: Resolved
• Support Namespace Level quota for Volume: Resolved
• Support Namespace Level quota for bucket: Resolved
• add storage space quota doc: Resolved
• The usage value needs to be adjusted after client crash: Resolved
• support quota on key prefix: Open
• Refactor OzoneQuota to make it easy to support more quota type: Resolved
• Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket: Resolved
• Volume Namespace:`usedNamespace` works by `ozone sh vol info`: Resolved
• Volume Namespace: check usedNamespace when create bucket: Resolved
• Volume Namespace: support clear namespace quota: Resolved
• Bucket Namespace: add namespaceQuotaUsage and update it when create and delete key in a bucket: Resolved
• Bucket Namespace: check usedNamespace when create key in bucket: Resolved
• Bucket Namespace: support clear namespace quota: Resolved
• CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility.: Resolved
• Old bucket needs to be accessible after the cluster was upgraded to the Quota version.: Resolved
• Update QUOTA_IN_COUNTS to quota in namespace: Resolved
• If volume's quota is enabled then bucket's quota cannot be cleared: Resolved
• Add namespace quota to documentation: Resolved
• Handle illegalArgumentException in Volume/Bucket Quota requests: Resolved
• Should remove allocated quota when open key are cleaned up: Resolved
• Update CountQuota to NamesapceQuota: Resolved
• Fixed client set quota with 0: Resolved
• Fix when update quota the usedBytes and namespace will become to 0: Resolved
• Add warning log when old volume and bucket set quota: Resolved
• OzoneQuota change BYTES to B: Resolved
• Allow Default Space Quota Unit as Bytes: Resolved
• Update usedBytes only when commit key.: Resolved
• S3MultipartUpload support update bucket usedNamespace.: Resolved