Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-541

Ozone Quota support.

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Target Version/s:

      Description

      Create a volume with just 1 MB as quota

      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh volume create --quota=1MB --user=root /hive
      2018-09-23 02:10:11,283 [main] INFO - Creating Volume: hive, with root as owner and quota set to 1048576 bytes.
      

      Now create a bucket and put a big key greater than 1MB in the bucket

      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh bucket create /hive/bucket1
      2018-09-23 02:10:38,003 [main] INFO - Creating Bucket: hive/bucket1, with Versioning false and Storage Type set to DISK
      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ls -l ../../ozone-0.3.0-SNAPSHOT.tar.gz
      -rw-r--r-- 1 root root 165903437 Sep 21 13:16 ../../ozone-0.3.0-SNAPSHOT.tar.gz
      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key put /hive/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
      volume/bucket/key name required in putKey
      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key put /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz ../../ozone-0.3.0-SNAPSHOT.tar.gz
      [root@ctr-e138-1518143905142-481027-01-000002 bin]# ./ozone sh key info /hive/bucket1/ozone-0.3.0-SNAPSHOT.tar.gz
      {
      "version" : 0,
      "md5hash" : null,
      "createdOn" : "Sun, 23 Sep 2018 02:13:02 GMT",
      "modifiedOn" : "Sun, 23 Sep 2018 02:13:08 GMT",
      "size" : 165903437,
      "keyName" : "ozone-0.3.0-SNAPSHOT.tar.gz",
      "keyLocations" : [ {
      "containerID" : 2,
      "localID" : 100772661343420416,
      "length" : 134217728,
      "offset" : 0
      }, {
      "containerID" : 3,
      "localID" : 100772661661007873,
      "length" : 31685709,
      "offset" : 0
      } ]
      }

      It was able to put a 165 MB file on a volume with just 1MB quota.

       

      Currently Ozone haven't support Quota, So I think this should be a new feature .
      The design document can be referred to the attachment. (design google docs)

        Attachments

        Issue Links

        1.
        Ozone sh volume client support quota option. Sub-task Resolved mingchao zhao

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 49h
        Actions
        2.
        Volume space: add quotaUsageInBytes and update it when create and delete key Sub-task Resolved mingchao zhao   Actions
        3.
        Add protobuf field to support quota. Sub-task Closed Simon Su   Actions
        4.
        Volume space: check quotaUsageInBytes when write key Sub-task Resolved mingchao zhao   Actions
        5.
        Ozone sh bucket client support quota option. Sub-task Resolved mingchao zhao   Actions
        6.
        Bucket space: Supports clearing spaceQuota Sub-task Resolved mingchao zhao   Actions
        7.
        Volume space: Supports clearing spaceQuota Sub-task Resolved mingchao zhao   Actions
        8.
        Bucket space: add usedBytes and update it when create and delete key Sub-task Resolved mingchao zhao   Actions
        9.
        Bucket space: check quotaUsageInBytes when write key Sub-task Resolved mingchao zhao   Actions
        10.
        Space Quota: bucket quota not exceeded, volume needn’t check. Sub-task Resolved mingchao zhao   Actions
        11.
        Support Namespace Level quota for Volume Sub-task Resolved Rui Wang   Actions
        12.
        Support Namespace Level quota for bucket Sub-task Resolved Rui Wang   Actions
        13.
        add storage space quota doc Sub-task Resolved mingchao zhao   Actions
        14.
        The usage value needs to be adjusted after client crash Sub-task Open Unassigned   Actions
        15.
        support quota on key prefix Sub-task Open Unassigned   Actions
        16.
        Refactor OzoneQuota to make it easy to support more quota type Sub-task Resolved YI-CHEN WANG   Actions
        17.
        Volume namespace: add namespaceQuotaUsage and update it when create and delete bucket Sub-task Resolved Rui Wang   Actions
        18.
        Volume Namespace:`usedNamespace` works by `ozone sh vol info` Sub-task In Progress Rui Wang   Actions
        19.
        Volume Namespace: check usedNamespace when create bucket Sub-task Resolved Rui Wang   Actions
        20.
        Volume Namespace: support clear namespace quota Sub-task Resolved Rui Wang   Actions
        21.
        Bucket Namespace: add namespaceQuotaUsage and update it when create and delete key in a bucket Sub-task Resolved Rui Wang   Actions
        22.
        Bucket Namespace: check usedNamespace when create key in bucket Sub-task Resolved Rui Wang   Actions
        23.
        Bucket Namespace: support clear namespace quota Sub-task Resolved Rui Wang   Actions
        24.
        CLI flag --quota should default to 'spaceQuota' to preserve backward compatibility. Sub-task Resolved Ayush Saxena   Actions
        25.
        Old bucket needs to be accessible after the cluster was upgraded to the Quota version. Sub-task Resolved mingchao zhao   Actions
        26.
        Update QUOTA_IN_COUNTS to quota in namespace Sub-task Resolved Rui Wang   Actions
        27.
        If volume's quota is enabled then bucket's quota cannot be cleared Sub-task Resolved mingchao zhao   Actions
        28.
        Add namespace quota to documentation Sub-task Resolved Rui Wang   Actions
        29.
        Handle illegalArgumentException in Volume/Bucket Quota requests Sub-task Resolved Bharat Viswanadham   Actions
        30.
        Should remove allocated quota when open key are cleaned up Sub-task Open Rui Wang   Actions
        31.
        Update CountQuota to NamesapceQuota Sub-task Resolved Rui Wang   Actions
        32.
        Fixed client set quota with 0 Sub-task Resolved mingchao zhao   Actions
        33.
        Fix when update quota the usedBytes and namespace will become to 0 Sub-task Resolved mingchao zhao   Actions
        34.
        Add warning log when old volume and bucket set quota Sub-task Resolved mingchao zhao   Actions
        35.
        OzoneQuota change BYTES to B Sub-task Resolved mingchao zhao   Actions
        36.
        Allow Default Space Quota Unit as Bytes Sub-task Resolved Rui Wang   Actions

          Activity

            People

            • Assignee:
              micahzhao mingchao zhao
              Reporter:
              nmaheshwari Namit Maheshwari

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Time Spent - 145h Remaining Estimate - 120h
                120h
                Logged:
                Time Spent - 145h Remaining Estimate - 120h
                145h

                  Issue deployment