Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22648

Snapshot TTL

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0, 1.5.0, 2.3.0
    • Component/s: snapshots
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Feature: Take a Snapshot With TTL for auto-cleanup

      Attribute:
      1. TTL
           - Specify TTL in sec while creating snapshot. e.g. snapshot 'mytable', 'snapshot1234', {TTL => 86400} (snapshot to be auto-cleaned after 24 hr)

      Configs:
      1. Default Snapshot TTL:
           - FOREVER by default
           - User specified Default TTL(sec) with config: hbase.master.snapshot.ttl

      2. If Snapshot cleanup is supposed to be stopped due to some snapshot restore activity, disable it with config:
           - hbase.master.cleaner.snapshot.disable: "true"
          With this config, HMaster needs restart just like any other hbase-site config.


      For more details, see the section "Take a Snapshot With TTL" in the HBase Reference Guide.
      Show
      Feature: Take a Snapshot With TTL for auto-cleanup Attribute: 1. TTL      - Specify TTL in sec while creating snapshot. e.g. snapshot 'mytable', 'snapshot1234', {TTL => 86400} (snapshot to be auto-cleaned after 24 hr) Configs: 1. Default Snapshot TTL:      - FOREVER by default      - User specified Default TTL(sec) with config: hbase.master.snapshot.ttl 2. If Snapshot cleanup is supposed to be stopped due to some snapshot restore activity, disable it with config:      - hbase.master.cleaner.snapshot.disable: "true"     With this config, HMaster needs restart just like any other hbase-site config. For more details, see the section "Take a Snapshot With TTL" in the HBase Reference Guide.

      Description

      Snapshots have a lifecycle that is independent from the table from which they are created. Although data in a table may be stored with TTL the data files containing them become frozen by the snapshot. Space consumed by expired cells will not be reclaimed by normal table housekeeping like compaction. While this is expected it can be inconvenient at scale. When many snapshots are under management and the data in various tables is expired by TTL some notion of optional TTL (and optional default TTL) for snapshots could be useful. It will help prevent the accumulation of junk files by automatically dropping the snapshot after the assigned TTL, making their data files eligible for cleaning. More comprehensive snapshot lifecycle management may be considered in the future but this one case is expected to be immediately useful given TTls on data are commonly applied for similar convenience. 

        Attachments

        1. Screen Shot 2019-07-16 at 11.06.03 AM.png
          289 kB
          Viraj Jasani
        2. Screen Shot 2019-07-10 at 9.06.36 PM.png
          251 kB
          Viraj Jasani
        3. Screen Shot 2019-07-10 at 8.52.30 PM.png
          171 kB
          Viraj Jasani
        4. Screen Shot 2019-07-10 at 8.49.13 PM.png
          247 kB
          Viraj Jasani
        5. HBASE-22648-master-v8.patch
          46 kB
          Viraj Jasani
        6. HBASE-22648-master-v6.patch
          104 kB
          Viraj Jasani
        7. HBASE-22648-master-v5.patch
          89 kB
          Viraj Jasani
        8. HBASE-22648-master-v4.patch
          71 kB
          Viraj Jasani
        9. HBASE-22648-master-v3.patch
          59 kB
          Viraj Jasani
        10. HBASE-22648-master-v2.patch
          53 kB
          Viraj Jasani
        11. HBASE-22648-master.patch
          37 kB
          Viraj Jasani
        12. HBASE-22648-branch-2.patch
          47 kB
          Andrew Kyle Purtell
        13. HBASE-22648-branch-1.patch
          51 kB
          Andrew Kyle Purtell
        14. HBASE-22648-branch-1.patch
          51 kB
          Andrew Kyle Purtell

          Issue Links

            Activity

              People

              • Assignee:
                vjasani Viraj Jasani
                Reporter:
                apurtell Andrew Kyle Purtell
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: