Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5547

Don't delete HFiles when in "backup mode"

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.95.0
    • None
    • None
    • Reviewed
    • Hide
      All HFiles are now automatically archived to the configured "hbase.table.archive.directory". HFiles are cleaned similarly to HLogs - a cleaner delegate chain that is instantiated as per the configured classes. Similar to hlog cleaners, a TimeToLiveHFileCleaner is used by default, and specified under the "hbase.master.hfilecleaner.plugins" configuration key.

      In unifying the two cleaner interfaces, the new configuration parameter to use the TimeToLiveHLogCleaner is:
        <property>
          <name>hbase.master.logcleaner.plugins</name>
          <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner</value>
       </property>

      It is still enabled by default, so nothing needs to be changed if you are not modifying the logcleaners.
      Show
      All HFiles are now automatically archived to the configured "hbase.table.archive.directory". HFiles are cleaned similarly to HLogs - a cleaner delegate chain that is instantiated as per the configured classes. Similar to hlog cleaners, a TimeToLiveHFileCleaner is used by default, and specified under the "hbase.master.hfilecleaner.plugins" configuration key. In unifying the two cleaner interfaces, the new configuration parameter to use the TimeToLiveHLogCleaner is:   <property>     <name>hbase.master.logcleaner.plugins</name>     <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner</value>  </property> It is still enabled by default, so nothing needs to be changed if you are not modifying the logcleaners.

    Description

      This came up in a discussion I had with Stack.
      It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either:
      1. rename HFiles to be delete to <file>.bck
      2. rename the HFiles into a special directory
      3. rename them to a general trash directory (which would not need to be tied to backup mode).

      That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those).

      #1 makes cleanup a bit harder.

      Attachments

        1. 5547.addendum-v3
          7 kB
          Ted Yu
        2. 5547-addendum-v4.txt
          0.7 kB
          Ted Yu
        3. 5547-v12.txt
          184 kB
          Ted Yu
        4. 5547-v16.txt
          196 kB
          Ted Yu
        5. hbase-5447-v8.patch
          168 kB
          Jesse Yates
        6. hbase-5447-v8.patch
          168 kB
          Ted Yu
        7. hbase-5547-0.94-backport-v0.patch
          147 kB
          Jesse Yates
        8. hbase-5547-v9.patch
          185 kB
          Jesse Yates
        9. java_HBASE-5547_v13.patch
          190 kB
          Jesse Yates
        10. java_HBASE-5547_v14.patch
          194 kB
          Jesse Yates
        11. java_HBASE-5547_v15.patch
          196 kB
          Ted Yu
        12. java_HBASE-5547_v4.patch
          195 kB
          Jesse Yates
        13. java_HBASE-5547_v5.patch
          195 kB
          Jesse Yates
        14. java_HBASE-5547_v6.patch
          195 kB
          Jesse Yates
        15. java_HBASE-5547_v7.patch
          162 kB
          Jesse Yates
        16. java_HBASE-5547.addendum
          7 kB
          Jesse Yates
        17. java_HBASE-5547.addendum-v1
          13 kB
          Jesse Yates
        18. java_HBASE-5547.addendum-v2
          7 kB
          Jesse Yates

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              jesse_yates Jesse Yates
              larsh Lars Hofhansl
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: