Details

    • Type: Task Task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Feature Description

      This is a subtask of HBase-7912 to support FULL backup/restore, and will complete the following function:

      Backup Restore example
      /* backup from sourcecluster to targetcluster                                  */
      /* if no table name specified, all tables from source cluster will be backuped */
      [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
      
      /* restore on targetcluser, this is a local restore                                             */
      /* backup_1396650096738 - backup image name                                                     */
      /* t1_dn,etc are the original table names. All tables will be restored if not specified         */
      /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
      [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
      
      /* restore from targetcluster back to source cluster, this is a remote restore
      [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
      

      Detail layout and frame work for the next jiras

      The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of HBase-7912 as described below:

      • bin/hbase : end-user command line interface to invoke BackupClient and RestoreClient
      • BackupClient.java : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
        • create incremental backup
        • cancel an ongoing backup
        • delete an exisitng backup image
        • describe the detailed informaiton of backup image
        • show history of all successful backups
        • show the status of the latest backup request
        • convert incremental backup WAL files into HFiles. either on-the-fly during create or after create
        • merge backup image
        • stop backup a table of existing backup image
        • show tables of a backup image
      • BackupCommands.java : a place to keep all the command usages and options
      • BackupManager.java : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
      • BackupHandler.java : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
        • timestamps info will be recorded in ZK
        • carry on incremental backup.
        • update backup progress
        • set flags of status
        • build up backupManifest file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
        • clean up after failed backup
        • clean up after cancelled backup
        • allow on-the-fly convert during incremental backup
      • BackupContext.java : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
      • BackupCopier.java : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
      • BackupExcpetion.java: to handle exception from backup/restore
      • BackupManifest.java : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
      • BackupStatus.java : encapsulate backup status at table level during backup progress
      • BackupUtil.java : utility methods during backup process
      • RestoreClient.java : 'main' entry for restore operations. This patch will only support 'full' backup.
      • RestoreUtil.java : utility methods during restore process
      • ExportSnapshot.java : remove 'final' so that another class SnapshotCopy.java can extends from it
      • SnapshotCopy.java : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
      • BackupRestoreConstants.java : add the constants used by backup/restore code.
      • HBackupFilesystem.java : the filesystem related api used by BackupClient and RestoreClient.

      Global log roll

      currently a customized one under org.apache.hadoop.hbase.backup.master and org.apache.hadoop.hbase.backup.regionserver
      HBASE-11148 is opened to provide a general 'global log roll', and fullbackup code will be modified to use the general 'global log roll' later once HBase-11148 is accepted by the community.

      Interface

      • currently, the code is under hbase-sever because it already contain a package name called 'backup'. If move to hbase-client, the pom file has to be updated to include more dependency
      • currently invoke through script bin/hbase as CLI interface. One advantage is easy to embed into a linux sh script
      1. HBASE-10900-trunk-v4.patch
        217 kB
        Demai Ni
      2. HBASE-10900-trunk-v3.patch
        160 kB
        Demai Ni
      3. HBASE-10900-trunk-v2.patch
        160 kB
        Demai Ni
      4. HBASE-10900-fullbackup-trunk-v1.patch
        151 kB
        Demai Ni

        Issue Links

          Activity

          Demai Ni created issue -
          Demai Ni made changes -
          Field Original Value New Value
          Link This issue is part of HBASE-7912 [ HBASE-7912 ]
          Demai Ni made changes -
          Description This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore.

          {code:title=Backup Restore example|borderStyle=solid}
          $hbase backup create full /user/biadmin/backup1 t1_dn
          $hbase backup create full hdfs://hostname.hbase.com:9000/user/biadmin/backup1 t2_dn
          $hbase restore /user/biadmin/backup1/ backup_1396206203119 t1_dn t1_dn_restore
          {code}

          h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *HConstants.java* : add the constants used by backup/restore code.

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          Hide
          Demai Ni added a comment -

          attached the patch to solicit inputs. I am in process to add the testcases. And look forward suggestion to improve the patch

          Show
          Demai Ni added a comment - attached the patch to solicit inputs. I am in process to add the testcases. And look forward suggestion to improve the patch
          Demai Ni made changes -
          Attachment HBASE-10900-fullbackup-trunk-v1.patch [ 12639450 ]
          Demai Ni made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Matteo Bertozzi added a comment -

          could you post that on review board https://reviews.apache.org/

          Show
          Matteo Bertozzi added a comment - could you post that on review board https://reviews.apache.org/
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639450/HBASE-10900-fullbackup-trunk-v1.patch
          against trunk revision .
          ATTACHMENT ID: 12639450

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause mvn compile goal to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + "Unsupported filesystem scheme found in the backup target url, please check the url to make sure no typo in it";
          + "Usage: hbase backup create <type> <backup_root_path> [tables] [-s name] [-convert] [-silent]\n"
          + + " tables If no tables (\"\") are specified, all tables are backed up. Otherwise it is a\n"
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/.");
          + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel.");
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been created under /complete/.");
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/.");
          + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel.");
          + LOG.error("Failed to cancel the backup " + backupContext.getBackupId() + " due to " + msg);
          + LOG.debug("Manifest file has been persisted to " + this.tableBackupDir != null ? this.tableBackupDir

          -1 site. The patch appears to cause mvn site goal to fail.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9233//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9233//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639450/HBASE-10900-fullbackup-trunk-v1.patch against trunk revision . ATTACHMENT ID: 12639450 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc . The javadoc tool did not generate any warning messages. -1 javac . The patch appears to cause mvn compile goal to fail. -1 findbugs . The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + "Unsupported filesystem scheme found in the backup target url, please check the url to make sure no typo in it"; + "Usage: hbase backup create <type> <backup_root_path> [tables] [-s name] [-convert] [-silent] \n" + + " tables If no tables (\"\") are specified, all tables are backed up. Otherwise it is a\n" + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/."); + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel."); + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been created under /complete/."); + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/."); + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel."); + LOG.error("Failed to cancel the backup " + backupContext.getBackupId() + " due to " + msg); + LOG.debug("Manifest file has been persisted to " + this.tableBackupDir != null ? this.tableBackupDir -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9233//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9233//console This message is automatically generated.
          Hide
          Demai Ni added a comment -

          Matteo, thanks for the suggestion. I created the review at https://reviews.apache.org/r/20176/

          Show
          Demai Ni added a comment - Matteo, thanks for the suggestion. I created the review at https://reviews.apache.org/r/20176/
          Hide
          Ted Yu added a comment -

          For the first question above, I think hbase-client module would be Okay.

          I don't have strong opinion for second question.

          Show
          Ted Yu added a comment - For the first question above, I think hbase-client module would be Okay. I don't have strong opinion for second question.
          Hide
          Demai Ni added a comment -

          Ted Yu, thanks. I can move the code to hbase-client module and within a new 'backup' package there.

          Ted Yu and Matteo Bertozzi, many thanks for your input through review board. I am going through them right now, and will incorporate in the next version of the patch.

          Demai

          Show
          Demai Ni added a comment - Ted Yu , thanks. I can move the code to hbase-client module and within a new 'backup' package there. Ted Yu and Matteo Bertozzi , many thanks for your input through review board. I am going through them right now, and will incorporate in the next version of the patch. Demai
          Hide
          stack added a comment -

          Demai Ni Is this a client-side only facility? A tool? Do you have hooks on the serverside (to ensure preservation of any needed WALs recovering?) Thanks.

          Show
          stack added a comment - Demai Ni Is this a client-side only facility? A tool? Do you have hooks on the serverside (to ensure preservation of any needed WALs recovering?) Thanks.
          Hide
          Demai Ni added a comment -

          stack, most of the code will be on client-side, we use zookeeper to keep the serverside info.

          Also there are a couple enhancements : Jerry's global flush HBase-10926 and global logroll(jira not open yet) built on top of Richard's HBase-9426 . We have a offline WalPlayer to convert HLog to HFiles, I am looking at this one right now, which is also depending on HBASE-8073

          Show
          Demai Ni added a comment - stack , most of the code will be on client-side, we use zookeeper to keep the serverside info. Also there are a couple enhancements : Jerry's global flush HBase-10926 and global logroll(jira not open yet) built on top of Richard's HBase-9426 . We have a offline WalPlayer to convert HLog to HFiles, I am looking at this one right now, which is also depending on HBASE-8073
          Hide
          Matteo Bertozzi added a comment -

          most of the code will be on client-side

          should we aim to merge the code in the server?
          with some refactor I guess we can merge the backup and snapshot code, and expose the "take full backup" and "take delta backup" as rpc.

          Also there are a couple enhancements : Jerry's global flush HBase-10926 and global logroll(jira not open yet)

          I guess the code for keeping the WALS is something like this

          • STEP-1: Call the Global Log-Roll
          • STEP-2: Copy Each log in the log directory
            how do you ensure that the log is not remove between STEP-1 and STEP-2?
            also not having connection on the server side, means that you are making a copy of the WAL and not just keeping a "reference". Don't you want to force a copy just on export?
          Show
          Matteo Bertozzi added a comment - most of the code will be on client-side should we aim to merge the code in the server? with some refactor I guess we can merge the backup and snapshot code, and expose the "take full backup" and "take delta backup" as rpc. Also there are a couple enhancements : Jerry's global flush HBase-10926 and global logroll(jira not open yet) I guess the code for keeping the WALS is something like this STEP-1: Call the Global Log-Roll STEP-2: Copy Each log in the log directory how do you ensure that the log is not remove between STEP-1 and STEP-2? also not having connection on the server side, means that you are making a copy of the WAL and not just keeping a "reference". Don't you want to force a copy just on export?
          Hide
          Jerry He added a comment -

          Hi, Matteo Bertozzi
          Another piece is an extra logcleaner plugin, which helps to hold the logs that we need to keep.
          The logs are not copied out in a full backup. Snapshot has all the full backup data.
          The full backup process includes a global log roll and a global snapshot. (These two procedures probably can be combined.)
          The global log roll is to record the starting of new logs for later incremental/delta backup: new log entries will be kept to be included in future incremental backup.

          Show
          Jerry He added a comment - Hi, Matteo Bertozzi Another piece is an extra logcleaner plugin, which helps to hold the logs that we need to keep. The logs are not copied out in a full backup. Snapshot has all the full backup data. The full backup process includes a global log roll and a global snapshot. (These two procedures probably can be combined.) The global log roll is to record the starting of new logs for later incremental/delta backup: new log entries will be kept to be included in future incremental backup.
          Hide
          Demai Ni added a comment -

          Matteo Bertozzi, I posted my response to your comments on review board last Friday, but forgot to publish them. a rookie mistake. Sorry about that. Many thanks for your input. about your comments on zookeeper, I will need to do some more study to address it. I am aware of some efforts recently through several Jiras to move away from zookeeper. .. Demai

          Show
          Demai Ni added a comment - Matteo Bertozzi , I posted my response to your comments on review board last Friday, but forgot to publish them. a rookie mistake. Sorry about that. Many thanks for your input. about your comments on zookeeper, I will need to do some more study to address it. I am aware of some efforts recently through several Jiras to move away from zookeeper. .. Demai
          Hide
          Demai Ni added a comment -

          Thanks for the comments from review board. I addressed some of them, and upload v2 patch here.

          I will open another jira for incremental backup, which will help to show whole picture of this solution. so I will switch my time to work on that for a few days. Then come back to address other review comments in this jira

          Show
          Demai Ni added a comment - Thanks for the comments from review board. I addressed some of them, and upload v2 patch here. I will open another jira for incremental backup, which will help to show whole picture of this solution. so I will switch my time to work on that for a few days. Then come back to address other review comments in this jira
          Demai Ni made changes -
          Attachment HBASE-10900-trunk-v2.patch [ 12642095 ]
          Demai Ni made changes -
          Description h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *HConstants.java* : add the constants used by backup/restore code.

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *BackupRestoreConstants.java* : add the constants used by backup/restore code.
          * *HBackupFilesystem.java* : the filesystem related api used by BackupClient and RestoreClient.

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          Demai Ni made changes -
          Link This issue is required by HBASE-11085 [ HBASE-11085 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642095/HBASE-10900-trunk-v2.patch
          against trunk revision .
          ATTACHMENT ID: 12642095

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 34 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 18 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + "Usage: hbase backup create <type> <backup_root_path> [tables] [-s name] [-convert] [-silent]\n"
          + + " tables If no tables (\"\") are specified, all tables are backed up. Otherwise it is a\n"
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/.");
          + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel.");
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been created under /complete/.");
          + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/.");
          + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel.");
          + LOG.error("Failed to cancel the backup " + backupContext.getBackupId() + " due to " + msg);
          + LOG.debug("Manifest file has been persisted to " + this.tableBackupDir != null ? this.tableBackupDir
          + protected static ZooKeeperWatcher getZKWatcher() throws ZooKeeperConnectionException, IOException {

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642095/HBASE-10900-trunk-v2.patch against trunk revision . ATTACHMENT ID: 12642095 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc . The javadoc tool appears to have generated 34 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 18 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + "Usage: hbase backup create <type> <backup_root_path> [tables] [-s name] [-convert] [-silent] \n" + + " tables If no tables (\"\") are specified, all tables are backed up. Otherwise it is a\n" + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/."); + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel."); + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been created under /complete/."); + LOG.debug("Backup znode " + backupContext.getBackupId() + " has been removed from /ongoing/."); + + " does not exists, should have already been deleted from /ongoing/ during cleanup for cancel."); + LOG.error("Failed to cancel the backup " + backupContext.getBackupId() + " due to " + msg); + LOG.debug("Manifest file has been persisted to " + this.tableBackupDir != null ? this.tableBackupDir + protected static ZooKeeperWatcher getZKWatcher() throws ZooKeeperConnectionException, IOException { +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9409//console This message is automatically generated.
          Enis Soztutar made changes -
          Fix Version/s 0.99.0 [ 12325675 ]
          Fix Version/s 1.0.0 [ 12325852 ]
          Hide
          Demai Ni added a comment -

          attached v3, which to clean up most of the javadoc warning, findingbug warning, and the long statement.

          there are still a couple findingbug warning related with static methods, which will be addressed later.

          Show
          Demai Ni added a comment - attached v3, which to clean up most of the javadoc warning, findingbug warning, and the long statement. there are still a couple findingbug warning related with static methods, which will be addressed later.
          Demai Ni made changes -
          Attachment HBASE-10900-trunk-v3.patch [ 12642392 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642392/HBASE-10900-trunk-v3.patch
          against trunk revision .
          ATTACHMENT ID: 12642392

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 4 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + + " -overwrite With this option, restore overwrites to the existing table if there's any in\n"
          + + " The restore dependencies can be checked by using \"-check\" option,\n"
          + + " or using \"hbase backup describe\" command. Without this option, only\n"
          + throw new IOException("Existing table found in target while no \"-overwrite\" option found");

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642392/HBASE-10900-trunk-v3.patch against trunk revision . ATTACHMENT ID: 12642392 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc . The javadoc tool appears to have generated 4 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + + " -overwrite With this option, restore overwrites to the existing table if there's any in\n" + + " The restore dependencies can be checked by using \"-check\" option,\n" + + " or using \"hbase backup describe\" command. Without this option, only\n" + throw new IOException("Existing table found in target while no \"-overwrite\" option found"); +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9418//console This message is automatically generated.
          Demai Ni made changes -
          Link This issue requires HBASE-10926 [ HBASE-10926 ]
          Hide
          Demai Ni added a comment -

          Attached v4 version for trunk, which includes
          1) UT (thanks to Enoch Hsu)
          2) small interface change to use Snapshot Manifest
          3) plugged in a customized 'global log roll' to record info into ZK. A general 'global log roll' jira will be opened a bit later. and Fullbackup code will depend on it

          Demai

          Show
          Demai Ni added a comment - Attached v4 version for trunk, which includes 1) UT (thanks to Enoch Hsu ) 2) small interface change to use Snapshot Manifest 3) plugged in a customized 'global log roll' to record info into ZK. A general 'global log roll' jira will be opened a bit later. and Fullbackup code will depend on it Demai
          Demai Ni made changes -
          Attachment HBASE-10900-trunk-v4.patch [ 12644142 ]
          Demai Ni made changes -
          Description h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *BackupRestoreConstants.java* : add the constants used by backup/restore code.
          * *HBackupFilesystem.java* : the filesystem related api used by BackupClient and RestoreClient.

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *BackupRestoreConstants.java* : add the constants used by backup/restore code.
          * *HBackupFilesystem.java* : the filesystem related api used by BackupClient and RestoreClient.

          h2. Global log roll
          currently a customized one under *org.apache.hadoop.hbase.backup.master* and *org.apache.hadoop.hbase.backup.regionserver*
          A seperated jira will opened to provide a general 'global log roll', and fullbackup code will be modified to use the general 'global log roll' later

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12644142/HBASE-10900-trunk-v4.patch
          against trunk revision .
          ATTACHMENT ID: 12644142

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 14 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 7 warning messages.

          -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + + " -overwrite With this option, restore overwrites to the existing table if there's any in\n"
          + + " The restore dependencies can be checked by using \"-check\" option,\n"
          + + " or using \"hbase backup describe\" command. Without this option, only\n"
          + throw new IOException("Existing table found in target while no \"-overwrite\" option found");

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:

          -1 core zombie tests. There are 1 zombie test(s):

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644142/HBASE-10900-trunk-v4.patch against trunk revision . ATTACHMENT ID: 12644142 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 14 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 7 warning messages. -1 findbugs . The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + + " -overwrite With this option, restore overwrites to the existing table if there's any in\n" + + " The restore dependencies can be checked by using \"-check\" option,\n" + + " or using \"hbase backup describe\" command. Without this option, only\n" + throw new IOException("Existing table found in target while no \"-overwrite\" option found"); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9491//console This message is automatically generated.
          Demai Ni made changes -
          Description h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *BackupRestoreConstants.java* : add the constants used by backup/restore code.
          * *HBackupFilesystem.java* : the filesystem related api used by BackupClient and RestoreClient.

          h2. Global log roll
          currently a customized one under *org.apache.hadoop.hbase.backup.master* and *org.apache.hadoop.hbase.backup.regionserver*
          A seperated jira will opened to provide a general 'global log roll', and fullbackup code will be modified to use the general 'global log roll' later

          h2. Questions for your inputs
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. Another place is *hbase-client*. Any prefereence?
          * Is bin/hbase the right place as CLI interface? another place would be hbase shell.


          h2. Feature Description
          This is a subtask of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL backup/restore, and will complete the following function:
          {code:title=Backup Restore example|borderStyle=solid}
          /* backup from sourcecluster to targetcluster */
          /* if no table name specified, all tables from source cluster will be backuped */
          [sourcecluster]$ hbase backup create full hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn

          /* restore on targetcluser, this is a local restore */
          /* backup_1396650096738 - backup image name */
          /* t1_dn,etc are the original table names. All tables will be restored if not specified */
          /* t1_dn_restore, etc. are the restored table. if not specified, orginal table name will be used*/
          [targetcluster]$ hbase restore /userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore

          /* restore from targetcluster back to source cluster, this is a remote restore
          [sourcecluster]$ hbase restore hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738 t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
          {code}

          h2. Detail layout and frame work for the next jiras
          The patch is a wrapper of the existing snapshot and exportSnapshot, and will use as the base framework for the over-all solution of [HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described below:
          * *bin/hbase* : end-user command line interface to invoke BackupClient and RestoreClient
          * *BackupClient.java* : 'main' entry for backup operations. This patch will only support 'full' backup. In future jiras, will support:
          ** *create* incremental backup
          ** *cancel* an ongoing backup
          ** *delete* an exisitng backup image
          ** *describe* the detailed informaiton of backup image
          ** show *history* of all successful backups
          ** show the *status* of the latest backup request
          ** *convert* incremental backup WAL files into HFiles. either on-the-fly during create or after create
          ** *merge* backup image
          ** *stop* backup a table of existing backup image
          ** *show* tables of a backup image
          * *BackupCommands.java* : a place to keep all the command usages and options
          * *BackupManager.java* : handle backup requests on server-side, create BACKUP ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be used for future incremental backup (not included in this jira). Create BackupContext and DispatchRequest.
          * *BackupHandler.java* : in this patch, it is a wrapper of snapshot and exportsnapshot. In future jiras,
          ** *timestamps* info will be recorded in ZK
          ** carry on *incremental* backup.
          ** update backup *progress*
          ** set flags of *status*
          ** build up *backupManifest* file(in this jira only limited info for fullback. later on, timestamps and dependency of multipl backup images are also recorded here)
          ** clean up after *failed* backup
          ** clean up after *cancelled* backup
          ** allow on-the-fly *convert* during incremental backup
          * *BackupContext.java* : encapsulate backup information like backup ID, table names, directory info, phase, TimeStamps of backup progress, size of data, ancestor info, etc.
          * *BackupCopier.java* : the copying operation. Later on, to support progress report and mapper estimation; and extends DisCp for progress updating to ZK during backup.
          * *BackupExcpetion.java*: to handle exception from backup/restore
          * *BackupManifest.java* : encapsulate all the backup image information. The manifest info will be bundled as manifest file together with data. So that each backup image will contain all the info needed for restore.
          * *BackupStatus.java* : encapsulate backup status at table level during backup progress
          * *BackupUtil.java* : utility methods during backup process
          * *RestoreClient.java* : 'main' entry for restore operations. This patch will only support 'full' backup.
          * *RestoreUtil.java* : utility methods during restore process
          * *ExportSnapshot.java* : remove 'final' so that another class SnapshotCopy.java can extends from it
          * *SnapshotCopy.java* : only a wrapper at this moment. But will be extended to keep track progress(maybe should implemented in ExportSnapshot directly?)
          * *BackupRestoreConstants.java* : add the constants used by backup/restore code.
          * *HBackupFilesystem.java* : the filesystem related api used by BackupClient and RestoreClient.

          h2. Global log roll
          currently a customized one under *org.apache.hadoop.hbase.backup.master* and *org.apache.hadoop.hbase.backup.regionserver*
          [HBASE-11148|https://issues.apache.org/jira/browse/HBASE-11148] is opened to provide a general 'global log roll', and fullbackup code will be modified to use the general 'global log roll' later once HBase-11148 is accepted by the community.

          h2. Interface
          * currently, the code is under *hbase-sever* because it already contain a package name called 'backup'. If move to *hbase-client*, the pom file has to be updated to include more dependency
          * currently invoke through script bin/hbase as CLI interface. One advantage is easy to embed into a linux sh script


          Demai Ni made changes -
          Link This issue requires HBASE-11148 [ HBASE-11148 ]
          Demai Ni made changes -
          Link This issue is required by HBASE-11172 [ HBASE-11172 ]
          Demai Ni made changes -
          Link This issue is required by HBASE-11173 [ HBASE-11173 ]
          Demai Ni made changes -
          Link This issue is required by HBASE-11174 [ HBASE-11174 ]
          Enoch Hsu made changes -
          Link This issue is depended upon by HBASE-11181 [ HBASE-11181 ]
          Enis Soztutar made changes -
          Fix Version/s 0.99.1 [ 12328551 ]
          Fix Version/s 0.99.0 [ 12325675 ]
          Enis Soztutar made changes -
          Fix Version/s 0.99.2 [ 12328822 ]
          Fix Version/s 0.99.1 [ 12328551 ]
          Enis Soztutar made changes -
          Fix Version/s 1.1.0 [ 12329043 ]
          Fix Version/s 0.99.2 [ 12328822 ]
          Hide
          ramkrishna.s.vasudevan added a comment -

          Demai Ni, Jerry He
          Any reason why this JIRA is not updated for some time now? Are you guys looking to get this in still? Just asking out of curiosity.

          Show
          ramkrishna.s.vasudevan added a comment - Demai Ni , Jerry He Any reason why this JIRA is not updated for some time now? Are you guys looking to get this in still? Just asking out of curiosity.
          Hide
          Demai Ni added a comment -

          ramkrishna.s.vasudevan, thanks for the ping. After chatted with several folks in hbase community, the plan is to build a stand-alone utility in github(or some place similar) instead of pushing the large portion of code into hbase core code. I think Jerry Heis still planning to get it done.

          Show
          Demai Ni added a comment - ramkrishna.s.vasudevan , thanks for the ping. After chatted with several folks in hbase community, the plan is to build a stand-alone utility in github(or some place similar) instead of pushing the large portion of code into hbase core code. I think Jerry He is still planning to get it done.
          Hide
          Andrew Purtell added a comment -

          Then we should resolve this issue.

          Show
          Andrew Purtell added a comment - Then we should resolve this issue.
          Hide
          Demai Ni added a comment -

          Andrew Purtell, sounds the right way to go.

          Jerry He, any objections? If not, I will go ahead resolve the jiras under my name as not fix.

          Show
          Demai Ni added a comment - Andrew Purtell , sounds the right way to go. Jerry He , any objections? If not, I will go ahead resolve the jiras under my name as not fix.
          Demai Ni made changes -
          Assignee Demai Ni [ nidmhbase ]
          Fix Version/s 1.1.0 [ 12329043 ]
          Hide
          Demai Ni added a comment -

          Due to personal reason, I can't work directly to contribute back to open source community at this moment. Put this jira as 'unassigned', and remove fixed version

          hopefully, someone can pick it up. or Or my situation may change and I can continue to work on this.

          Thanks... Demai

          Show
          Demai Ni added a comment - Due to personal reason, I can't work directly to contribute back to open source community at this moment. Put this jira as 'unassigned', and remove fixed version hopefully, someone can pick it up. or Or my situation may change and I can continue to work on this. Thanks... Demai
          Hide
          Jerry He added a comment -

          Had offline discussion with Demai Ni and Andrew Purtell.
          Will keep the JIRA open so that people from our team or other people can work in it in the future.
          Thanks, Demai, Andrew.

          Show
          Jerry He added a comment - Had offline discussion with Demai Ni and Andrew Purtell . Will keep the JIRA open so that people from our team or other people can work in it in the future. Thanks, Demai, Andrew.

            People

            • Assignee:
              Unassigned
              Reporter:
              Demai Ni
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development