Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18135

Track file archival for low latency space quota with snapshots

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Changes the manner in which file space consumption is reported to the Master for the purposes of space quota tracking to reduce the latency in which system space utilization is observed. This will have a positive effect in how quickly HBase will react to changes in filesystem usage related to file archiving.
      Show
      Changes the manner in which file space consumption is reported to the Master for the purposes of space quota tracking to reduce the latency in which system space utilization is observed. This will have a positive effect in how quickly HBase will react to changes in filesystem usage related to file archiving.

      Description

      Related to the work proposed on HBASE-17748 and building on the same idea as HBASE-18133, we can make the space quota tracking for HBase snapshots faster to respond.

      When snapshots are in play, the location of a file (whether in the data or archive directory) plays a factor in the realized size of a table. Like flushes, compactions, etc, moving files from the data directory to the archive directory is done by the RegionServer. We can hook into this call and send the necessary information to the Master so that it can more quickly update the size of a table when there are snapshots in play.

      This will require the RegionServer to report the full coordinates of the file being moved (table+region+family+file) so that the SnapshotQuotaObserverChore running in the master can avoid HDFS lookups in partial or total to compute the location of a Region's hfiles.

      This may also require some refactoring of the SnapshotQuotaObserverChore to de-couple the receipt of these file archival reports from RegionServers (e.g. HRegionFileSystem.removeStoreFiles(..), and the Master processing the sizes of snapshots.

        Attachments

        1. HBASE-18135.001.patch
          223 kB
          Josh Elser
        2. HBASE-18135.002.patch
          111 kB
          Josh Elser
        3. HBASE-18135.004.patch
          112 kB
          Josh Elser
        4. HBASE-18135.005.patch
          111 kB
          Josh Elser

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                elserj Josh Elser
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: