Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14123

HBase Backup/Restore Phase 2

    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.0.0-alpha-1
    • None
    • None
    • Reviewed
    • Hide
      Vladimir Rodionov, Feb 2017

      *** Summary of work HBASE-14123

      The new feature introduces new command-line extensions to the hbase command and, from the client side, is accessible through command-line only
      Operations:
      * Create full backup on a list of tables or backup set
      * Create incremental backup image for table list or backup set
      * Restore list of tables from a given backup image
      * Show current backup progress
      * Delete backup image and all related images
      * Show history of backups
      * Backup set operations: create backup set, add/remove table to/from backup set, etc

      In the current implementation, the feature is already usable, meaning that users can backup tables and restore them using provided command-line tools. Both: full and incremental backups are supported.
      This work is based on original work of IBM team (HBASE-7912). The full list of JIRAs included in this mega patch can be found in three umbrella JIRAs: HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones made it into the patch)

      *** What are the remaining work items

      All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
      They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
      Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.

      ***** BLOCKER

      * HBASE-14417 Incremental backup and bulk loading ( Patch available)
      * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
      * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables (Patch available)
      * HBASE-17133 Backup documentation
      * HBASE-15227 Fault tolerance support

      ***** CRITICAL

      * HBASE-16465 Disable split/merges during backup

      We have umbrella JIRA (HBASE-14414) to track all the remaining work
      All the BLOCKER and CRITICAL JIRAs currently in open state will be implemented by 2.0 release time. Some MAJOR too, but it depends on resource availability
      The former development branch (HBASE-7912) is obsolete and will be closed/deleted after the merge.
      We want backup to be a GA feature in 2.0
      We are going to support full backward compatibility for backup tool in 2.0 and onwards.

      **** Configuration

      Backup is disabled, by default. To enable it, the following configuration properties must be added to hbase-site.xml:

      hbase.backup.enable=true
      hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner
      hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
      hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager


      **** Known Issues

      * Bulk loaded data is not visible for incremental backup (will be resolved by 2.0 release)
      * After first backup sessions, WAL files will be accumulated in old WAL directory until the next backup sessions. Make sure, that you either run backups regularly or disable backups in the system.
      * No fault tolerance yet, some backup failures may result in corrupted backup data and will require manual intervention (will be resolved by 2.0 release)


      Show
      Vladimir Rodionov, Feb 2017 *** Summary of work HBASE-14123 The new feature introduces new command-line extensions to the hbase command and, from the client side, is accessible through command-line only Operations: * Create full backup on a list of tables or backup set * Create incremental backup image for table list or backup set * Restore list of tables from a given backup image * Show current backup progress * Delete backup image and all related images * Show history of backups * Backup set operations: create backup set, add/remove table to/from backup set, etc In the current implementation, the feature is already usable, meaning that users can backup tables and restore them using provided command-line tools. Both: full and incremental backups are supported. This work is based on original work of IBM team ( HBASE-7912 ). The full list of JIRAs included in this mega patch can be found in three umbrella JIRAs: HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all resolved ones made it into the patch) *** What are the remaining work items All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414 . They are split into 3 groups: BLOCKER, CRITICAL, MAJOR Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release. ***** BLOCKER * HBASE-14417 Incremental backup and bulk loading ( Patch available) * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables (Patch available) * HBASE-17133 Backup documentation * HBASE-15227 Fault tolerance support ***** CRITICAL * HBASE-16465 Disable split/merges during backup We have umbrella JIRA ( HBASE-14414 ) to track all the remaining work All the BLOCKER and CRITICAL JIRAs currently in open state will be implemented by 2.0 release time. Some MAJOR too, but it depends on resource availability The former development branch ( HBASE-7912 ) is obsolete and will be closed/deleted after the merge. We want backup to be a GA feature in 2.0 We are going to support full backward compatibility for backup tool in 2.0 and onwards. **** Configuration Backup is disabled, by default. To enable it, the following configuration properties must be added to hbase-site.xml: hbase.backup.enable=true hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager **** Known Issues * Bulk loaded data is not visible for incremental backup (will be resolved by 2.0 release) * After first backup sessions, WAL files will be accumulated in old WAL directory until the next backup sessions. Make sure, that you either run backups regularly or disable backups in the system. * No fault tolerance yet, some backup failures may result in corrupted backup data and will require manual intervention (will be resolved by 2.0 release)

    Description

      Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.

      Attachments

        1. HBASE-14123-v9.patch
          119 kB
          Vladimir Rodionov
        2. HBASE-14123-v7.patch
          111 kB
          Vladimir Rodionov
        3. HBASE-14123-v6.patch
          100 kB
          Vladimir Rodionov
        4. HBASE-14123-v5.patch
          418 kB
          Vladimir Rodionov
        5. HBASE-14123-v4.patch
          396 kB
          Vladimir Rodionov
        6. HBASE-14123-v3.patch
          392 kB
          Vladimir Rodionov
        7. HBASE-14123-v2.patch
          389 kB
          Vladimir Rodionov
        8. HBASE-14123-v16.patch
          243 kB
          Vladimir Rodionov
        9. HBASE-14123-v15.patch
          224 kB
          Vladimir Rodionov
        10. HBASE-14123-v13.patch
          201 kB
          Vladimir Rodionov
        11. HBASE-14123-v12.patch
          192 kB
          Vladimir Rodionov
        12. HBASE-14123-v11.patch
          158 kB
          Vladimir Rodionov
        13. HBASE-14123-v10.patch
          125 kB
          Vladimir Rodionov
        14. HBASE-14123-v1.patch
          374 kB
          Vladimir Rodionov
        15. HBASE-14123-for-7912-v6.patch
          496 kB
          Vladimir Rodionov
        16. HBASE-14123-for-7912-v1.patch
          249 kB
          Vladimir Rodionov
        17. Backup-restoreinHBase2.0 (4).pdf
          84 kB
          Vladimir Rodionov
        18. Backup-restoreinHBase2.0 (3).pdf
          84 kB
          Vladimir Rodionov
        19. Backup-restoreinHBase2.0 (1).pdf
          37 kB
          Vladimir Rodionov
        20. Backup-restoreinHBase2.0.pdf
          37 kB
          Vladimir Rodionov
        21. 14123-v14.txt
          195 kB
          Ted Yu
        22. 14123-master.v9.txt
          1.09 MB
          Ted Yu
        23. 14123-master.v8.txt
          1.09 MB
          Ted Yu
        24. 14123-master.v7.txt
          1.09 MB
          Ted Yu
        25. 14123-master.v6.txt
          1.09 MB
          Ted Yu
        26. 14123-master.v5.txt
          1.09 MB
          Ted Yu
        27. 14123-master.v38.txt
          965 kB
          Vladimir Rodionov
        28. 14123-master.v37.txt
          965 kB
          Vladimir Rodionov
        29. 14123-master.v36.txt
          964 kB
          Vladimir Rodionov
        30. 14123-master.v35.txt
          964 kB
          Vladimir Rodionov
        31. 14123-master.v34.txt
          964 kB
          Vladimir Rodionov
        32. 14123-master.v33.txt
          962 kB
          Vladimir Rodionov
        33. 14123-master.v32.txt
          947 kB
          Vladimir Rodionov
        34. 14123-master.v31.txt
          949 kB
          Vladimir Rodionov
        35. 14123-master.v30.txt
          945 kB
          Vladimir Rodionov
        36. 14123-master.v3.txt
          1.09 MB
          Ted Yu
        37. 14123-master.v29.full.txt
          1.01 MB
          Vladimir Rodionov
        38. 14123-master.v28.txt
          1.40 MB
          Vladimir Rodionov
        39. 14123-master.v27.txt
          1.33 MB
          Ted Yu
        40. 14123-master.v25.txt
          1.32 MB
          Ted Yu
        41. 14123-master.v24.txt
          1.32 MB
          Ted Yu
        42. 14123-master.v21.txt
          1.30 MB
          Ted Yu
        43. 14123-master.v20.txt
          1.30 MB
          Ted Yu
        44. 14123-master.v2.txt
          1.09 MB
          Ted Yu
        45. 14123-master.v19.txt
          1.28 MB
          Ted Yu
        46. 14123-master.v18.txt
          1.28 MB
          Ted Yu
        47. 14123-master.v17.txt
          1.28 MB
          Ted Yu
        48. 14123-master.v16.txt
          1.13 MB
          Ted Yu
        49. 14123-master.v15.txt
          1.13 MB
          Ted Yu
        50. 14123-master.v14.txt
          1.13 MB
          Ted Yu
        51. 14123.master.v61.patch
          886 kB
          Vladimir Rodionov
        52. 14123.master.v60.patch
          886 kB
          Vladimir Rodionov
        53. 14123.master.v59.patch
          875 kB
          Vladimir Rodionov
        54. 14123.master.v58.patch
          858 kB
          Vladimir Rodionov
        55. 14123.master.v57.patch
          862 kB
          Vladimir Rodionov
        56. 14123.master.v56.patch
          858 kB
          Vladimir Rodionov
        57. 14123.master.v54.patch
          858 kB
          Vladimir Rodionov
        58. 14123.master.v52.patch
          871 kB
          Vladimir Rodionov
        59. 14123.master.v51.patch
          869 kB
          Vladimir Rodionov
        60. 14123.master.v50.patch
          871 kB
          Vladimir Rodionov
        61. 14123.master.v49.patch
          878 kB
          Vladimir Rodionov
        62. 14123.master.v48.patch
          887 kB
          Vladimir Rodionov
        63. 14123.master.v46.patch
          887 kB
          Vladimir Rodionov
        64. 14123.master.v45.patch
          887 kB
          Vladimir Rodionov
        65. 14123.master.v44.patch
          926 kB
          Vladimir Rodionov
        66. 14123.master.v42.patch
          909 kB
          Vladimir Rodionov
        67. 14123.master.v41.patch
          965 kB
          Vladimir Rodionov
        68. 14123.master.v40.patch
          962 kB
          Vladimir Rodionov
        69. 14123.master.v39.patch
          962 kB
          Vladimir Rodionov

        Issue Links

          Activity

            People

              vrodionov Vladimir Rodionov
              vrodionov Vladimir Rodionov
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: