Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-4282

Redesign quota feature

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 3.8.0
    • Component/s: quota
    • Labels:
      None

      Description

      Quota Use Case:

      Generally in a big data solution deployment multiple services (hdfs, yarn, hbase etc.) use single Zookeeper cluster. So it is very important to ensure fare usage by all services. Sometime services unintentionally, mainly because of faulty behavior, create many znodes and impact the overall reliability of the ZooKeeper service. To ensure the faire usage quota feature is required. But this is the only use case there are many other use cases for quota feature.

      Current Problems:

      1. Currently, user can set quota by updating znode “/zookeeper/quota/nodepath”, or using setquota/delquota in CLI command.
        This makes the quota setting infective
        Currently any user can set/delete quota, which is not proper, it should be admin operation
      2. User is allowed to modify zookeeper system paths like /zookeeper/quota. These are internal to zookeeper should not be allowed to modify.
      3. Generally services create single top level znode in Zookeeper like /hbase and create all required znode under it.
        It is better if it is configurable who can create top level znodes to controll ZooKeeper usage.
      4. After ZOOKEEPER-231, there two kinds quota enforcement limits 1. Hard limit 2. Soft limit.
        I think there should be only limit. When enforce quota is enabled that limits becomes the hard limit otherwise it is soft limit same as old feature, just logs warnings.

      Proposed Solution

      1. Add setQuota and deleteQuota admin APIs. Add listQuota normal user API
        Modify quota cli commands to use these APIs instead of directory modifying ZooKeeper system path /zookeeper/quota/
      2. Protect ZooKeeper system paths from outside modification. System should only be readable from outside
      3. Expose configuration to set ACL for root system znode.
        After this, at the time of ZooKeeper service deployment administrator can create top level znode for a service and set quota. This way we can control overall ZooKeeper usage
      4. Revert some of the changes in ZOOKEEPER-231 and move to single quota limit

        Attachments

          Activity

            People

            • Assignee:
              arshad.mohammad Mohammad Arshad
              Reporter:
              arshad.mohammad Mohammad Arshad

              Dates

              • Created:
                Updated:

                Issue deployment