Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14417

Incremental backup and bulk loading

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 2.0.0
    • None
    • Reviewed

    Description

      Currently, incremental backup is based on WAL files. Bulk data loading bypasses WALs for obvious reasons, breaking incremental backups. The only way to continue backups after bulk loading is to create new full backup of a table. This may not be feasible for customers who do bulk loading regularly (say, every day).

      Here is the review board (out of date):
      https://reviews.apache.org/r/54258/

      In order not to miss the hfiles which are loaded into region directories in a situation where postBulkLoadHFile() hook is not called (bulk load being interrupted), we record hfile names thru preCommitStoreFile() hook.
      At time of incremental backup, we check the presence of such hfiles. If they are present, they become part of the incremental backup image.

      Here is review board:
      https://reviews.apache.org/r/57790/

      Google doc for design:
      https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE

      Attachments

        1. 14417.v1.txt
          65 kB
          Ted Yu
        2. 14417.v11.txt
          52 kB
          Ted Yu
        3. 14417.v13.txt
          58 kB
          Ted Yu
        4. 14417.v2.txt
          69 kB
          Ted Yu
        5. 14417.v21.txt
          74 kB
          Ted Yu
        6. 14417.v23.txt
          74 kB
          Ted Yu
        7. 14417.v24.txt
          81 kB
          Ted Yu
        8. 14417.v25.txt
          83 kB
          Ted Yu
        9. 14417.v6.txt
          43 kB
          Ted Yu
        10. 14417-tbl-ext.v10.txt
          66 kB
          Ted Yu
        11. 14417-tbl-ext.v11.txt
          66 kB
          Ted Yu
        12. 14417-tbl-ext.v14.txt
          72 kB
          Ted Yu
        13. 14417-tbl-ext.v18.txt
          73 kB
          Ted Yu
        14. 14417-tbl-ext.v19.txt
          71 kB
          Ted Yu
        15. 14417-tbl-ext.v20.txt
          71 kB
          Ted Yu
        16. 14417-tbl-ext.v21.txt
          73 kB
          Ted Yu
        17. 14417-tbl-ext.v22.txt
          72 kB
          Ted Yu
        18. 14417-tbl-ext.v23.txt
          73 kB
          Ted Yu
        19. 14417-tbl-ext.v24.txt
          73 kB
          Ted Yu
        20. 14417-tbl-ext.v9.txt
          67 kB
          Ted Yu

        Issue Links

          Activity

            People

              yuzhihong@gmail.com Ted Yu
              vrodionov Vladimir Rodionov
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: