Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16961

FileSystem Quotas

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • None
    • Reviewed

    Description

      Umbrella issue for tracking the filesystem utilization of HBase data, defining quotas on that utilization, and enforcement when utilization exceeds the limits of the quota.

      At a high level: we can define quotas on tables and namespaces. Region size is computed by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling up to table and namespace sizes. Defined quotas in the quota table are evaluated given the computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed to take some action to limit any further filesystem growth by that table/namespace.

      Discuss: https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E
      Design Doc: http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing

      Attachments

        Issue Links

        1.
        Build client Java API and client protobuf messages Sub-task Resolved Josh Elser Actions
        2.
        Implement storage/retrieval of filesystem-use quotas into quota table Sub-task Resolved Josh Elser Actions
        3.
        [Master] Collect table use information from RegionServers Sub-task Resolved Josh Elser Actions
        4.
        [Master] Analyze table use reports and update quota violations Sub-task Resolved Josh Elser Actions
        5.
        [Master] Inform RegionServers on size quota violations Sub-task Resolved Josh Elser Actions
        6.
        [RegionServer] Compute region filesystem space use and report to Master Sub-task Resolved Josh Elser Actions
        7.
        [RegionServer] Implement enforcement of quota violation policies Sub-task Resolved Josh Elser Actions
        8.
        [Master] Report quotas and active violations on Master UI and JMX metrics Sub-task Resolved Josh Elser Actions
        9.
        Update book for filesystem use quotas Sub-task Resolved Josh Elser Actions
        10.
        [Shell] Support space quota get/set via the shell Sub-task Resolved Josh Elser Actions
        11.
        Missing functionality to remove space quota Sub-task Resolved Josh Elser Actions
        12.
        Expand on shell commands for detailed insight Sub-task Resolved Josh Elser Actions
        13.
        Automatically delete quota when table is deleted Sub-task Resolved Josh Elser Actions
        14.
        Avoid sending FSUtilization reports to master when quota support is not enabled Sub-task Resolved Josh Elser Actions
        15.
        Table quota not taking precedence over namespace quota Sub-task Resolved Josh Elser Actions
        16.
        SecureBulkLoadEndpoint not implemented Sub-task Resolved Josh Elser Actions
        17.
        HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException Sub-task Resolved Ted Yu Actions
        18.
        Expire region reports in the HMaster Sub-task Resolved Josh Elser Actions
        19.
        Tweak chore delay/period defaults Sub-task Resolved Josh Elser Actions
        20.
        Remove lingering "violation" in favor of the accurate "snapshot" Sub-task Resolved Josh Elser Actions
        21.
        Commit Reviewboard comments from Vlad Sub-task Resolved Josh Elser Actions
        22.
        Remove stability annotation from public audience-marked classes Sub-task Resolved Josh Elser Actions
        23.
        Enabled Master observer to delete quotas on table deletion by default Sub-task Resolved Josh Elser Actions
        24.
        Investigate hbase superuser permissions in the face of quota violation Sub-task Resolved Josh Elser Actions
        25.
        Roll list_quota_violations into list_quota_snapshots Sub-task Resolved Josh Elser Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            elserj Josh Elser Assign to me
            elserj Josh Elser
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment