Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18133

Low-latency space quota size reports

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Presently space quota enforcement relies on RegionServers sending reports to the master about each Region that they host. This is done by periodically, reading the cached size of each HFile in each Region (which was ultimately computed from HDFS).

      This means that the Master is unaware of Region size growth until the the next time this chore in a RegionServer fires which is a fair amount of latency (a few minutes, by default). Operations like flushes, compactions, and bulk-loads are delayed even though the RegionServer is running those operations locally.

      Instead, we can create an API which these operations could invoke that would automatically update the size of the Region being operated on. For example, a successful flush can report that the size of a Region increased by the size of the flush. A compaction can subtract the size of the input files of the compaction and add in the size of the resulting file.

      This de-couples the computation of a Region's size from sending the Region sizes to the Master, allowing us to send reports more frequently, increasing the responsiveness of the cluster to size changes.

        Attachments

        1. HBASE-18133.009.patch
          117 kB
          Josh Elser
        2. HBASE-18133.008.patch
          118 kB
          Josh Elser
        3. HBASE-18133.007.patch
          114 kB
          Josh Elser
        4. HBASE-18133.006.patch
          113 kB
          Josh Elser
        5. HBASE-18133.005.patch
          113 kB
          Josh Elser
        6. HBASE-18133.004.patch
          113 kB
          Josh Elser
        7. HBASE-18133.003.patch
          113 kB
          Josh Elser
        8. HBASE-18133.002.patch
          111 kB
          Josh Elser
        9. HBASE-18133.001.patch
          111 kB
          Josh Elser

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                elserj Josh Elser
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: