Details

    • Hadoop Flags:
      Reviewed
    • Tags:
      0.96notable

      Description

      Continuation of HBASE-50 for the current trunk. Since the implementation has drastically changed, opening as a new ticket.

      1. Offline Snapshots.docx
        977 kB
        Jesse Yates
      2. Snapshots in HBase.docx
        454 kB
        Jesse Yates

        Issue Links

        1.
        Extract daemon thread factory from HTable into its own class Sub-task Closed Jesse Yates
         
        2.
        Extract HStore interface from Store Sub-task Closed Jesse Yates
         
        3.
        Fix hfile/log cleaning delegate method naming Sub-task Closed Jesse Yates
         
        4.
        Generic multi-thread/cross-process error handling framework Sub-task Closed Jesse Yates
         
        5.
        Distributed Three-Phase Commit framework. Sub-task Resolved Jesse Yates
         
        6.
        'Take a snapshot' interface Sub-task Closed Jesse Yates
         
        7.
        Snapshot Restore interface Sub-task Closed Matteo Bertozzi
         
        8.
        Export Snapshot Sub-task Closed Matteo Bertozzi
         
        9.
        Offline snapshots Sub-task Closed Jesse Yates
         
        10.
        Snapshot File Cleaners Sub-task Closed Jesse Yates
         
        11.
        Snapshots shell Sub-task Closed Matteo Bertozzi
         
        12.
        [snapshots] Refactor error handling to use javax.management Sub-task Resolved Jesse Yates
         
        13.
        Snapshot References Utils (FileSystem Visitor) Sub-task Closed Matteo Bertozzi
         
        14.
        [snapshots] Refactor snapshot file cleaner cache to use the Snapshot FileVisitor Sub-task Closed Matteo Bertozzi
         
        15.
        Foreign Exception framework v2 (simplifies and replaces HBASE-6571) Sub-task Closed Jonathan Hsieh
         
        16.
        Consolidate snapshot related classes into fewer packages Sub-task Closed Jonathan Hsieh
         
        17.
        Transition Offline Snapshots to ForeignExceptions Sub-task Closed Jonathan Hsieh
         
        18.
        HFileLink: Hardlink alternative for snapshot restore Sub-task Closed Matteo Bertozzi
         
        19.
        [brainstorm] "Restore" snapshots for HBase 0.96 Sub-task Closed Matteo Bertozzi
         
        20.
        Cleanup old snapshots on start Sub-task Closed Jesse Yates
         
        21.
        Add snapshot verification admin tool Sub-task Resolved Matteo Bertozzi
         
        22.
        Check for snapshot file cleaners on start Sub-task Closed Matteo Bertozzi
         
        23.
        Add snapshot information to hbase master webui Sub-task Closed Matteo Bertozzi
         
        24.
        Splitting a hfilelink causes region servers to go down. Sub-task Closed Jonathan Hsieh
         
        25.
        clone operation from HBaseAdmin can hang forever. Sub-task Closed Matteo Bertozzi
         
        26.
        Snapshot Info/Debug Tool Sub-task Closed Matteo Bertozzi
         
        27.
        Snapshot coprocessor and ACL security Sub-task Closed Matteo Bertozzi
         
        28.
        Snapshot branch 12/18 rebase broke TestSnapshotFromMaster#testSnapshotHFileArchiving Sub-task Closed Jonathan Hsieh
         
        29.
        HBASE-7365 Snapshot clone/restore followup Sub-task Closed Matteo Bertozzi
         
        30.
        ExportSnapshot mapper closes the FileSystem Sub-task Closed Matteo Bertozzi
         
        31.
        revisit hfilelink file name format. Sub-task Closed Matteo Bertozzi
         
        32.
        TestSnapshotExceptionSnare and TestWALReferenceTask missing test annotation failing TestCheckTestClasses Sub-task Closed Jonathan Hsieh
         
        33.
        TestSnapshotDescriptionUtils break compaction/scanner tests (EnvironmentEdge issue) Sub-task Closed Matteo Bertozzi
         
        34.
        Improve stack trace info dumped by ForeignExceptionSnare#rethrowException Sub-task Closed Jonathan Hsieh
         
        35.
        HFileLink should not use the configuration from the Filesystem Sub-task Closed Matteo Bertozzi
         
        36.
        Change ForeignException#receive(String, FE) to only be #receive(FE) Sub-task Closed Jonathan Hsieh
         
        37.
        HBASE-7423 snapshot followup Sub-task Closed Matteo Bertozzi
         
        38.
        Move ErrorHandling protos from hbase-server to hbase-protocol Sub-task Closed Jonathan Hsieh
         
        39.
        Enable Cleaners required for Snapshots by default Sub-task Closed Ted Yu
         
        40.
        Explicit message for not allowed snapshot on meta tables Sub-task Closed Matteo Bertozzi
         
        41.
        Fix Restore with schema changes Sub-task Closed Matteo Bertozzi
         
        42.
        Fix restore reference files Sub-task Closed Matteo Bertozzi
         
        43.
        Fix findbugs warnings in snapshot classes Sub-task Closed Ted Yu
         
        44.
        Fixes and cleanups Sub-task Closed Matteo Bertozzi
         
        45.
        Remove duplicated code from HFileLink Sub-task Closed Matteo Bertozzi
         
        46.
        Add table descriptor verification after snapshot restore Sub-task Closed Matteo Bertozzi
         
        47.
        Remove duplicated logFSTree() from TestRestoreFlushSnapshotFromClient Sub-task Closed Matteo Bertozzi
         
        48.
        CloneTableHandler notify completion too early Sub-task Closed Matteo Bertozzi
         
        49.
        Race in the Restore Archiving Sub-task Closed Matteo Bertozzi
         
        50.
        Safer table creation and deletion using .tmp dir Sub-task Closed Matteo Bertozzi
         

          Activity

          Hide
          Jesse Yates added a comment -

          Initial, very rough draft of the code is up on github: https://github.com/jyates/hbase/tree/snapshots. Latest commit message has the following description (which is a pretty good summary of the current status):

          This is based on the backup-hfile work in HBASE-5547, and is very different
          from the actual proposal (and code samples) in HBASE-50. The overall layout is
          more or less the same as the proposal, but they actual implementation has changed
          due to changes in the actual code as well as the current conventions.

          Currently, only 1 test is written and has not been tested (99% sure its not
          going to pass). However, the meat of the implementation is complete. There is
          still some work around listing and deleting of online snapshots and taking of
          snapshots for offlined tables, but this is trivial compared
          to the taking of snapshots for an online table and naturally falls out of the
          current implementation.

          Further, the export/import functionality for snapshots has not been completed, but will
          probably (again) be very similar to the work in HBASE-50. Currently Matteo Bertozzi is
          interested in working on this functionality, so I'm leaving that to him for the moment.

          NOTES:

          • I'm not very happy with the monitoring infrastructure I've put in place around
            keeping track of the different tasks and propagating errors from snapshots failing
            locally (from any of the various async threads) back to the rest of the nodes running
            the snapshot and vice-versa. Its feels overly complex and seems to be refactored repeatedly.
            Its on the list, but seemed not worth the effort of cleanup versus having something up.

          Planning on doing a writeup this weekend so people can actually have a chance at splunking through the changeset (which is non-trivial in size).

          Show
          Jesse Yates added a comment - Initial, very rough draft of the code is up on github: https://github.com/jyates/hbase/tree/snapshots . Latest commit message has the following description (which is a pretty good summary of the current status): This is based on the backup-hfile work in HBASE-5547 , and is very different from the actual proposal (and code samples) in HBASE-50 . The overall layout is more or less the same as the proposal, but they actual implementation has changed due to changes in the actual code as well as the current conventions. Currently, only 1 test is written and has not been tested (99% sure its not going to pass). However, the meat of the implementation is complete. There is still some work around listing and deleting of online snapshots and taking of snapshots for offlined tables, but this is trivial compared to the taking of snapshots for an online table and naturally falls out of the current implementation. Further, the export/import functionality for snapshots has not been completed, but will probably (again) be very similar to the work in HBASE-50 . Currently Matteo Bertozzi is interested in working on this functionality, so I'm leaving that to him for the moment. NOTES: I'm not very happy with the monitoring infrastructure I've put in place around keeping track of the different tasks and propagating errors from snapshots failing locally (from any of the various async threads) back to the rest of the nodes running the snapshot and vice-versa. Its feels overly complex and seems to be refactored repeatedly. Its on the list, but seemed not worth the effort of cleanup versus having something up. Planning on doing a writeup this weekend so people can actually have a chance at splunking through the changeset (which is non-trivial in size).
          Hide
          Jesse Yates added a comment -

          Attaching longer document detailing snapshots. It has both a general overview and a walkthrough of implementation. Its complementary to the docs on HBASE-50 and the code on github.

          In short, its distributed two-phase commit, where the prepare phase blocks all writes to all, enabled via a barrier node in zookeeper.

          Show
          Jesse Yates added a comment - Attaching longer document detailing snapshots. It has both a general overview and a walkthrough of implementation. Its complementary to the docs on HBASE-50 and the code on github. In short, its distributed two-phase commit, where the prepare phase blocks all writes to all, enabled via a barrier node in zookeeper.
          Hide
          Ted Yu added a comment -

          The design document is very good.
          Will get back to reviewing HBASE-5547 first.

          Show
          Ted Yu added a comment - The design document is very good. Will get back to reviewing HBASE-5547 first.
          Hide
          Jesse Yates added a comment -

          Adding updated documentation - realized it fudged a couple things when doing the testing (thanks for the hints Matteo!)

          Show
          Jesse Yates added a comment - Adding updated documentation - realized it fudged a couple things when doing the testing (thanks for the hints Matteo!)
          Hide
          Jimmy Xiang added a comment -

          I have a concern. Why should we do two phases? I think the prepare phase is not needed. We have row level atomicity. We don't need every region server to be on the same page. Since it is distributed, it is arguable about the meaning of point-in-time. That means it is hard to say it is consistent/inconsistent point-in-time.

          I think we each region server can try to create the snapshot at first. If anyone fails, partial snapshot can be just deleted.

          Show
          Jimmy Xiang added a comment - I have a concern. Why should we do two phases? I think the prepare phase is not needed. We have row level atomicity. We don't need every region server to be on the same page. Since it is distributed, it is arguable about the meaning of point-in-time. That means it is hard to say it is consistent/inconsistent point-in-time. I think we each region server can try to create the snapshot at first. If anyone fails, partial snapshot can be just deleted.
          Hide
          Ted Yu added a comment -

          I checked out project from github:

          # On branch snapshots
          nothing to commit (working directory clean)
          

          I got some compilation errors:

          [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java:[102,0] org.apache.hadoop.hbase.master.MockRegionServer is not abstract and does not override abstract method getRootDir() in org.apache.hadoop.hbase.regionserver.RegionServerServices
          [ERROR] 
          [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[340,12] cannot find symbol
          [ERROR] symbol  : method waitForFlushesAndCompactions()
          [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
          [ERROR] 
          [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[426,12] cannot find symbol
          [ERROR] symbol  : method waitForFlushesAndCompactions()
          [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
          [ERROR] 
          [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[507,12] cannot find symbol
          [ERROR] symbol  : method waitForFlushesAndCompactions()
          [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
          [ERROR] 
          [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java:[45,7] org.apache.hadoop.hbase.util.MockRegionServerServices is not abstract and does not override abstract method getRootDir() in org.apache.hadoop.hbase.regionserver.RegionServerServices
          
          Show
          Ted Yu added a comment - I checked out project from github: # On branch snapshots nothing to commit (working directory clean) I got some compilation errors: [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java:[102,0] org.apache.hadoop.hbase.master.MockRegionServer is not abstract and does not override abstract method getRootDir() in org.apache.hadoop.hbase.regionserver.RegionServerServices [ERROR] [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[340,12] cannot find symbol [ERROR] symbol : method waitForFlushesAndCompactions() [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion [ERROR] [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[426,12] cannot find symbol [ERROR] symbol : method waitForFlushesAndCompactions() [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion [ERROR] [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java:[507,12] cannot find symbol [ERROR] symbol : method waitForFlushesAndCompactions() [ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion [ERROR] [ERROR] /Users/zhihyu/snapshots/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java:[45,7] org.apache.hadoop.hbase.util.MockRegionServerServices is not abstract and does not override abstract method getRootDir() in org.apache.hadoop.hbase.regionserver.RegionServerServices
          Hide
          Ted Yu added a comment -

          Some files, such as RegionSnapshotPool, don't have license.

          For RegionSnapshotOperation.java:

            public void setStatusMonitor(RegionSnapshotOperationStatus monitor) {
              this.setStatus(new RegionSnapshotStatus(monitor));
          

          the method name doesn't seem to match its parameter.

          Will post more comments later.

          Show
          Ted Yu added a comment - Some files, such as RegionSnapshotPool, don't have license. For RegionSnapshotOperation.java: public void setStatusMonitor(RegionSnapshotOperationStatus monitor) { this .setStatus( new RegionSnapshotStatus(monitor)); the method name doesn't seem to match its parameter. Will post more comments later.
          Hide
          Jesse Yates added a comment -

          sorry, forgot to mention that the new "correct" branch is snapshots-r0. It compiles locally for me and gets most of the way through the test (TestSnapshotFromClient).Should solve most of the current issues That'll teach me to be overzealous with posting patches.

          Show
          Jesse Yates added a comment - sorry, forgot to mention that the new "correct" branch is snapshots-r0. It compiles locally for me and gets most of the way through the test (TestSnapshotFromClient).Should solve most of the current issues That'll teach me to be overzealous with posting patches.
          Hide
          gaojinchao added a comment -

          This is a very useful feature. :0

          Show
          gaojinchao added a comment - This is a very useful feature. :0
          Hide
          gaojinchao added a comment -

          Hi Jesse, Are you working this feature? I am interested in it. I will study your code.
          one question, When we are creating snapshots, Do we need stop the balance?

          Show
          gaojinchao added a comment - Hi Jesse, Are you working this feature? I am interested in it. I will study your code. one question, When we are creating snapshots, Do we need stop the balance?
          Hide
          ramkrishna.s.vasudevan added a comment -

          Nice doc Jesse.

          Show
          ramkrishna.s.vasudevan added a comment - Nice doc Jesse.
          Hide
          Jonathan Hsieh added a comment -

          Jesse,

          Thanks for the writeup, I find having a single doc with the design summary really helpful and ideally something we do for all major new features. I've read through the document carefully, let it steep for a few days, and had some design-level questions. I've skimmed HBASE-50 and will read more of the history more carefully later this evening.

          What is the read mechanism for snapshots like? Does the snapshot act like a read-only table or is there some special external mechanism needed to read the data from a snapshot? You mention having to rebuild in-memory state by replaying wals – is this a recovery situation or needed in normal reads?

          What is a representation of a snapshot look like in terms of META and file system contents? At some point we may get called upon to repair these, I want to make sure there are enough breadcrumbs for this to be possible.

          I'm still thinking about the two-phase part – I think it is necessary for marking success or initiating failure recovery, but I'm skeptical at the moment about why the barriering writes is necessary. How does this buy your more consistency? Aren't we still inconsistent at the prepare point now instead? Can we just write the special snapshotting hlog entry at initiation of prepare, allowing writes to continue, then adding data elsewhere (META) to mark success in commit? We could then have some compaction/flush time logic cleanup failed atttempt markers?

          Show
          Jonathan Hsieh added a comment - Jesse, Thanks for the writeup, I find having a single doc with the design summary really helpful and ideally something we do for all major new features. I've read through the document carefully, let it steep for a few days, and had some design-level questions. I've skimmed HBASE-50 and will read more of the history more carefully later this evening. What is the read mechanism for snapshots like? Does the snapshot act like a read-only table or is there some special external mechanism needed to read the data from a snapshot? You mention having to rebuild in-memory state by replaying wals – is this a recovery situation or needed in normal reads? What is a representation of a snapshot look like in terms of META and file system contents? At some point we may get called upon to repair these, I want to make sure there are enough breadcrumbs for this to be possible. I'm still thinking about the two-phase part – I think it is necessary for marking success or initiating failure recovery, but I'm skeptical at the moment about why the barriering writes is necessary. How does this buy your more consistency? Aren't we still inconsistent at the prepare point now instead? Can we just write the special snapshotting hlog entry at initiation of prepare, allowing writes to continue, then adding data elsewhere (META) to mark success in commit? We could then have some compaction/flush time logic cleanup failed atttempt markers?
          Hide
          Jesse Yates added a comment -

          @gaojinchao: I'm definitely still working on it - its just been a busy week, what with hbasecon, the hackathon and the rebase, this has been on the back burner. This week I'm planning to have a working first cut. Keep in mind that the code on github is a rough preview - definitely not the finished version, so no guarantees on polish or even correctness. That said, any feedback is appreciated.

          @Jon - I'm working on a thorough response, thanks for the questions.

          Show
          Jesse Yates added a comment - @gaojinchao: I'm definitely still working on it - its just been a busy week, what with hbasecon, the hackathon and the rebase, this has been on the back burner. This week I'm planning to have a working first cut. Keep in mind that the code on github is a rough preview - definitely not the finished version, so no guarantees on polish or even correctness. That said, any feedback is appreciated. @Jon - I'm working on a thorough response, thanks for the questions.
          Hide
          Jonathan Hsieh added a comment -

          Thanks Jesse,

          I haven't looked at code yet but I got a chance to read through the writeup in HBASE-50. One thing I like about it is that it breaks out the operations, pretty clearly describes the zk and hdfs layouts so that an admin, a tool, or someone with info at this particular level could use the feature and inspect the exposed zk and hdfs state for debugging.

          From the HBASE-50 doc (haven't looked at code yet), Li seems to have chosen or more clearly stated these design decisions:

          • hlog roll (which I believe does not trigger a flush) instead of special meta hlog marker (this might avoid write unavailability, seems simpler that the mechanism I suggested)
          • admin initiated snapshot and admin initiated restore operations as opposed to acting like a read only table. (not sure what happens to "newer" data after a restore, need to reread to see if it is in there, not sure about the cost to restore a snapshot)
          • I believe it also has an ability to read files directly from an MR job without having to go through HBase's get/put interface. Is that in scope for HBASE-6055?

          Jon.

          Show
          Jonathan Hsieh added a comment - Thanks Jesse, I haven't looked at code yet but I got a chance to read through the writeup in HBASE-50 . One thing I like about it is that it breaks out the operations, pretty clearly describes the zk and hdfs layouts so that an admin, a tool, or someone with info at this particular level could use the feature and inspect the exposed zk and hdfs state for debugging. From the HBASE-50 doc (haven't looked at code yet), Li seems to have chosen or more clearly stated these design decisions: hlog roll (which I believe does not trigger a flush) instead of special meta hlog marker (this might avoid write unavailability, seems simpler that the mechanism I suggested) admin initiated snapshot and admin initiated restore operations as opposed to acting like a read only table. (not sure what happens to "newer" data after a restore, need to reread to see if it is in there, not sure about the cost to restore a snapshot) I believe it also has an ability to read files directly from an MR job without having to go through HBase's get/put interface. Is that in scope for HBASE-6055 ? Jon.
          Hide
          Jesse Yates added a comment -

          I've recently had an existential crisis, of sorts, over snapshots. Triggered by both Jon's questions and some from Ian Varley, I've started to rethink the goal of snapshot. Initially, it was to take a globally consistent view of a single table. The question that Ian raised is, "Why are we enforcing stricter guarantees for a snapshot than for a scan?" In fact, a globally consistent view is something HBase explicitly doesn't support (if you do a put to two different tables, you have no real, system level guarantees of consistency).

          So does it really matter if we have an actual point in time? Everything in HBase is timestamped, which is considered the source of truth for a given Mutation. If we are doing a scan for the state of the table as of 12:15:05, we don't know if RS1 is 2 seconds before RS2 - as far as we care, its just the state at 12:15:05.

          This starts to break down a little bit when doing a Get for the latest version on a table. If RS1 is two seconds behind RS2 and we snapshot at 12:15:05, then we actually might not see all the change to RS1 in the snapshot. However, this doesn't really matter because you still wouldn't see that edit when looking at that "time". Things are happening so fast in HBase that the best we really need is just a "fuzzy" view of the state of the table.

          The upside to this is we can do the snapshot without taking any downtime on the table being snapshotted. I already discussed how to do this generally in the document, but it will have to be rewritten from the perspective of timestamped based snapshots (I'll move it to a google doc until we get a more finalized version).

          The only problem that has jumped out in multiple discussions of the timestamp based approach is that if you are using the timestamp for something other than the time (ala Facebook Messages) you might not be able to make use of snapshots. At Salesforce, I was planning on abusing timestamps as well, so that consideration will be made in the implementation (I'll go over how in another post).

          TL;DR global consistency doesn't matter for HBase since the timestamp is the source of truth - the only question is whether you believe the timestamp or not. I would posit that based on the design of HBase it has to be considered a source of truth.

          I'll respond in a bit with a more detailed design of how timestamp based snapshots differ from the point-in-time design, but in everything except how to deal with the memstore and WAL, it exactly the same. The way to handle the memstore was suggested by Ian Varley in that we basically use the memstore snapshot stuff with some rejiggering to wait a certain amount of time; for the WAL we can just use the meta edits that Jon recommends and that I've at least talked about IRL (if not in text).

          Show
          Jesse Yates added a comment - I've recently had an existential crisis, of sorts, over snapshots. Triggered by both Jon's questions and some from Ian Varley, I've started to rethink the goal of snapshot. Initially, it was to take a globally consistent view of a single table. The question that Ian raised is, "Why are we enforcing stricter guarantees for a snapshot than for a scan?" In fact, a globally consistent view is something HBase explicitly doesn't support (if you do a put to two different tables, you have no real, system level guarantees of consistency). So does it really matter if we have an actual point in time? Everything in HBase is timestamped, which is considered the source of truth for a given Mutation. If we are doing a scan for the state of the table as of 12:15:05, we don't know if RS1 is 2 seconds before RS2 - as far as we care, its just the state at 12:15:05. This starts to break down a little bit when doing a Get for the latest version on a table. If RS1 is two seconds behind RS2 and we snapshot at 12:15:05, then we actually might not see all the change to RS1 in the snapshot. However, this doesn't really matter because you still wouldn't see that edit when looking at that "time". Things are happening so fast in HBase that the best we really need is just a "fuzzy" view of the state of the table. The upside to this is we can do the snapshot without taking any downtime on the table being snapshotted. I already discussed how to do this generally in the document, but it will have to be rewritten from the perspective of timestamped based snapshots (I'll move it to a google doc until we get a more finalized version). The only problem that has jumped out in multiple discussions of the timestamp based approach is that if you are using the timestamp for something other than the time (ala Facebook Messages) you might not be able to make use of snapshots. At Salesforce, I was planning on abusing timestamps as well, so that consideration will be made in the implementation (I'll go over how in another post). TL;DR global consistency doesn't matter for HBase since the timestamp is the source of truth - the only question is whether you believe the timestamp or not. I would posit that based on the design of HBase it has to be considered a source of truth. I'll respond in a bit with a more detailed design of how timestamp based snapshots differ from the point-in-time design, but in everything except how to deal with the memstore and WAL, it exactly the same . The way to handle the memstore was suggested by Ian Varley in that we basically use the memstore snapshot stuff with some rejiggering to wait a certain amount of time; for the WAL we can just use the meta edits that Jon recommends and that I've at least talked about IRL (if not in text).
          Hide
          Jesse Yates added a comment - - edited

          (jon: I made a minor formatting tweak to make this easier to read the dir structure)

          But before a detailed description of how timestamp-based snapshots work internally, lets answer some comments!

          @Jon: I'll add more info to the document to cover this stuff, but for the moment, lets just get it out there.

          What is the read mechanism for snapshots like? Does the snapshot act like a read-only table or is there some special external mechanism needed to read the data from a snapshot? You mention having to rebuild in-memory state by replaying wals – is this a recovery situation or needed in normal reads?

          Its almost, but not quite like a table. Read of a snapshot is going to require an external tool but after hooking up the snapshot via the external tool, it should act just like a real table.

          Snapshots are intended to happen as fast as possible, to minimize downtime for the table. To enable that, we are just creating reference files in the snapshot directory. My vision is that once you take a snapshot, at some point (maybe weekly), you export the snapshot to a backup area. In the export you actually do the copy of the referenced files - you do a direct scan of the HFile (avoiding the top-level interface and going right to HDFS) and the WAL files. Then when you want to read the snapshot, you can just bulk-import the HFIles and replay the WAL files (with the WALPlayer this is relatively easy) to rebuild the state of the table at the time of the snapshot. Its not an exact copy (META isn't preserved), but all the actual data is there.

          The caveat here is since everything is references, one of the WAL files you reference may not actually have been closed (and therefore not readable). In the common case this won't happen, but if you snap and immediately export, its possible. In that case, you need to roll the WAL for the RS that haven't rolled them yet. However, this is in the export process, so a little latency there is tolerable, whereas avoiding this means adding latency to taking a snapshot - bad news bears.

          Keep in mind that the log files and hfiles will get regularly cleaned up. The former will be moved to the .oldlogs directory and periodically cleaned up and the latter get moved to the .archive directory (again with a parallel file hierarchy, as per HBASE-5547). If the snapshot goes to read the reference file, which tracks down to the original file and it doesn't find it, then it will need to lookup the same file in its respective archive directory. If its not there, then you are really hosed (except for the case mentioned in the doc about the WALs getting cleaned up by an aggressive log cleaner, which it is shown, is not a problem).

          Haven't gotten around to implementing this yet, but it seems reasonable to finish up (and I think Matteo was interested in working on that part).

          What is a representation of a snapshot look like in terms of META and file system contents?

          The way I see the implementation in the end is just a bunch of files in the /hbase/.snapshot directory. Like I mentioned above, the layout is very similar to the layout of a table.

          Lets look at an example of a table named "stuff" (snapshot names need to be valid directory names - same as a table or CF) and has column "column" which is hosted on servers rs-1 and rs-2. Originally, the file system will look something like (with license taken on file names - its not exact, I know, this is just an example) :

          /hbase/
          	.logs/
          		rs-1/
          			WAL-rs1-1
          			WAL-rs1-2
          		rs-2/
          			WAL-rs2-1
          			WAL-rs2-2
          	stuff/
          		.tableinfo
          		region1
          			column
          				region1-hfile-1
          		region2
          			column
          				region2-hfile-1
          

          The snapshot named "tuesday-at-nine", when completed, then just adds the following to the directory structure (or close enough):

          	.snapshot/
          		tuesday-at-nine/
          			.tableinfo
          			.snapshotinfo
          			.logs
          				rs-1/
          				WAL-rs1-1.reference
          				WAL-rs1-2.reference
          			rs-2/
          				WAL-rs2-1.reference
          				WAL-rs2-2.reference
          			stuff/
          				.tableinfo
          				region1
          					column
          						region1-hfile-1.reference
          				region2
          					column
          						region2-hfile-1.reference
          

          The only file here that isn't a reference here is the tableinfo since it is a pretty small file (generally), so a copy seemed more prudent over doing archiving on changes to the table info.

          The original implementation updated META with file references to do hbase-level hard links for the HFiles. AFter getting the original implementation working, I'm going to be ripping this piece out in favor of just doing an HFile cleaner and cleaner delegates (similar to logs) and then have a snapshot cleaner that reads of the FS for file references.

          At some point we may get called upon to repair these, I want to make sure there are enough breadcrumbs for this to be possible.

          How could that happen - hbase never has problems! (sarcasm)

          • hlog roll (which I believe does not trigger a flush) instead of special meta hlog marker (this might avoid write unavailability, seems simpler that the mechanism I suggested)

          The hlog marker is what I'm planning on doing for the timestamped based snapshot, which is going to be far safer than doing an HLog roll and provide less latency. With the roll, you need to not take any writes to the memstore between the roll and the end of the snapshot (otherwise you will lose edits). Doing meta edits into the HLog allows you to keep edits and not worry about it.

          admin initiated snapshot and admin initiated restore operations as opposed to acting like a read only table. (not sure what happens to "newer" data after a restore, need to reread to see if it is in there, not sure about the cost to restore a snapshot)

          Yup, right now its all handled from HBaseAdmin. Matteo was interested in working on the restore stuff, but depending on timing, I may end up picking up that work when I get the taking of a snapshot working. I think part of "snapshots" definitely includes getting back the state.

          I believe it also has an ability to read files directly from an MR job without having to go through HBase's get/put interface. Is that in scope for HBASE-6055?

          Absolutely in scope. It just didn't come up because I considered that part of the restore (which Matteo expressed interest). If you had to go through the high-level interface, then you would just use the procedure Lars talks about in his blog: http://hadoop-hbase.blogspot.com/2012/04/timestamp-consistent-backups-in-hbase.html

          The other notable change is that I'm building to support multiple snapshots concurrently. Its really a trivial change, so I don't think its too much feature creep, just a matter of using lists rather than a single item.

          How does this buy your more consistency? Aren't we still inconsistent at the prepare point now instead? Can we just write the special snapshotting hlog entry at initiation of prepare, allowing writes to continue, then adding data elsewhere (META) to mark success in commit? We could then have some compaction/flush time logic cleanup failed atttempt markers?

          See the above comment about timestamp based vs. point in time and the former being all that's necessary for HBase. This means we don't take downtime and end up with a 'fuzzy' snapshot in terms of global consistency, but is exact in terms of HBase delivered timestamps.

          The problem point-in-time snapshots overcomes is reaching distributed consensus while still trying to maintain availability and the ability to cross partitions. Since no one has figured out CAP and we are looking for consistency, we have to remove some availability to reach consensus. In this case, the agreement is over the state of the entire table, rather than per region server.

          Yes, this is strictly against the contract that we have on a Scan, but it is also in line with expectations people have on what a snapshot means. Any writes that are pending before the snapshot are allowed to commit, but any writes that reach the RS after the snapshot time cannot be included in the snapshot. I got a little overzealous in my reading of HBASE-50 and took it to mean global state, but after review the only way it would work within the constraints (no downtime) is to make it timestamp based.

          But why can't we get global consistency without taking downtime?

          Let's take your example of using an HLog edit to mark the start (and for ease, lets say the end as well - as long as its durable and recoverable, it doesn't matter if its WAL or META).

          Say we start a snapshot and send a message to all the RS (lets ignore ZK for the moment, to simplify things) that they should take a snapshot. So they write a marker into the HLog marking the start, create references as mentioned above, and then report to the master that they are done. When everyone is done, we then message each RS to commit the snapshot, which is just another entry into the WAL. Then in rebuilding the snapshot, they would just replay the WAL up to the start (assuming the end is found).

          How do we know though which writes arrived first on each RS if we just dump a write into the WAL? Ok, so then we need to wait for the MVCC read number to roll forward to when we got the snapshot notification before we can write an edit to the log - totally reasonable.

          However, the problem arises in attempting to get a global state of the table in a high-volume write environment. We have no guarantee that the "snapshot commit" notification reached each of the RS at the same time. And even if it did reach them at the same time, maybe there was some latency in getting the write number. Or the switch was a little wonky, or it just finishing up a GC (I could go on).

          Then we have a case where we don't actually have the snapshot as of the commit, but rather "at commit, plus or minus a bit" - not a clean snapshot (if we don't care about being exact then we can do a much faster, lower potential latency solution, the discussion of which is still coming, I promise). In a system that can take millions of writes a second, that is still a non-trivial amount of data that can change in a few milliseconds, no longer a true 'point in time'.

          The only way to get that global, consistent view is to remove the availability of the table for a short time so we know that the state is the same across all tables.

          Say we start a snapshot and the start indication doesn't reach the servers and get started at exactly the same time on all the servers, which, as explained above, is very likely. Then we let the servers commit any outstanding writes,but they don't get to take any new writes or a short time. In this time while they are waiting for writes to commit, we can then do all the snapshot preparation (referencing, table info copying). Once we are ready for the snapshot, we report back to the master and wait for the commit step. In this time we are still not taking writes. The key here is that for that short time, none of the servers are taking writes and that allows us to get a single point in time that no writes are committing (but they do get buffered on the server, they just can't change the system state).

          If we let writes commit, then how do we reach a state that we can agree on across all the servers? If you let the writes commit, you again don't have any assurances that the prepare or the commit message time is agreed to by all the servers. The table-level consistent state is somewhere between the prepare and commit, but it's not clear how one would find that point - I'm pretty sure we can't do this unless we have perfectly synchronized clocks, which is not really possible without a better understanding of quantum mechanics

          Block writes is a perhaps a bad phrase in this situation. In the current implementation, it buffers the writes as threads into the server, blocking on the updateLock. However, we can go with a "semi-blocking" version: writes still complete, but they aren't going to be visible until we roll forward to the snapshot MVCC number. This lets the writers complete (not affecting latency), but is going to affect read-modify-write and reader-to-writer comparison latency. However, as soon as we roll forward the MVCC, all those writes become visible, essentially catching back up to the current state. A slight modification to the WAL edits will need to be made to write the MVCC number so we can keep track of which writes are in/out of a snapshot, but that shouldn't be too hard (famous last words). You don't even need to modify all the WAL edits, just those made during the snapshot window, so the over the wire cost is still kept essentially the same, when amortized over the life of a table (for the standard use case).

          I'm looking at doing this once I get the simple version working - one step at a time. Moving to the timestamp based approach lets us keep taking writes but does so at the cost of global consistency in favor of local consistency and still uses the exact same infrastructure. The first patch I'll actually put on RB will be the timestamp based, but let me get the stop the world version going before going down a rabbit hole.

          The only thing we don't capture is if a writer makes a request to the RS before the snapshot is taken (by another client), but the write doesn't reach the server until after the RS hits the start barrier. From the global client perspective, this write should be in the snapshot, but that requires a single client or client-side write coordination (via a timestamp oracle). However, this is even worse coordination and creates even more constraints on the system where we currently have no coordination between clients (and I'm against adding any). So yes, we miss that edit, but that would be the case in a single-server database anyways without an external timestamp manager (to again distributed coordination between the client and server, though it can be done in a non-blocking manner). I'll mention some of this external coordination in the timestamp explanation.

          Show
          Jesse Yates added a comment - - edited (jon: I made a minor formatting tweak to make this easier to read the dir structure) But before a detailed description of how timestamp-based snapshots work internally, lets answer some comments! @Jon: I'll add more info to the document to cover this stuff, but for the moment, lets just get it out there. What is the read mechanism for snapshots like? Does the snapshot act like a read-only table or is there some special external mechanism needed to read the data from a snapshot? You mention having to rebuild in-memory state by replaying wals – is this a recovery situation or needed in normal reads? Its almost, but not quite like a table. Read of a snapshot is going to require an external tool but after hooking up the snapshot via the external tool, it should act just like a real table. Snapshots are intended to happen as fast as possible, to minimize downtime for the table. To enable that, we are just creating reference files in the snapshot directory. My vision is that once you take a snapshot, at some point (maybe weekly), you export the snapshot to a backup area. In the export you actually do the copy of the referenced files - you do a direct scan of the HFile (avoiding the top-level interface and going right to HDFS) and the WAL files. Then when you want to read the snapshot, you can just bulk-import the HFIles and replay the WAL files (with the WALPlayer this is relatively easy) to rebuild the state of the table at the time of the snapshot. Its not an exact copy (META isn't preserved), but all the actual data is there. The caveat here is since everything is references, one of the WAL files you reference may not actually have been closed (and therefore not readable). In the common case this won't happen, but if you snap and immediately export, its possible. In that case, you need to roll the WAL for the RS that haven't rolled them yet. However, this is in the export process, so a little latency there is tolerable, whereas avoiding this means adding latency to taking a snapshot - bad news bears. Keep in mind that the log files and hfiles will get regularly cleaned up. The former will be moved to the .oldlogs directory and periodically cleaned up and the latter get moved to the .archive directory (again with a parallel file hierarchy, as per HBASE-5547 ). If the snapshot goes to read the reference file, which tracks down to the original file and it doesn't find it, then it will need to lookup the same file in its respective archive directory. If its not there, then you are really hosed (except for the case mentioned in the doc about the WALs getting cleaned up by an aggressive log cleaner, which it is shown, is not a problem). Haven't gotten around to implementing this yet, but it seems reasonable to finish up (and I think Matteo was interested in working on that part). What is a representation of a snapshot look like in terms of META and file system contents? The way I see the implementation in the end is just a bunch of files in the /hbase/.snapshot directory. Like I mentioned above, the layout is very similar to the layout of a table. Lets look at an example of a table named "stuff" (snapshot names need to be valid directory names - same as a table or CF) and has column "column" which is hosted on servers rs-1 and rs-2. Originally, the file system will look something like (with license taken on file names - its not exact, I know, this is just an example) : /hbase/ .logs/ rs-1/ WAL-rs1-1 WAL-rs1-2 rs-2/ WAL-rs2-1 WAL-rs2-2 stuff/ .tableinfo region1 column region1-hfile-1 region2 column region2-hfile-1 The snapshot named "tuesday-at-nine", when completed, then just adds the following to the directory structure (or close enough): .snapshot/ tuesday-at-nine/ .tableinfo .snapshotinfo .logs rs-1/ WAL-rs1-1.reference WAL-rs1-2.reference rs-2/ WAL-rs2-1.reference WAL-rs2-2.reference stuff/ .tableinfo region1 column region1-hfile-1.reference region2 column region2-hfile-1.reference The only file here that isn't a reference here is the tableinfo since it is a pretty small file (generally), so a copy seemed more prudent over doing archiving on changes to the table info. The original implementation updated META with file references to do hbase-level hard links for the HFiles. AFter getting the original implementation working, I'm going to be ripping this piece out in favor of just doing an HFile cleaner and cleaner delegates (similar to logs) and then have a snapshot cleaner that reads of the FS for file references. At some point we may get called upon to repair these, I want to make sure there are enough breadcrumbs for this to be possible. How could that happen - hbase never has problems! (sarcasm) hlog roll (which I believe does not trigger a flush) instead of special meta hlog marker (this might avoid write unavailability, seems simpler that the mechanism I suggested) The hlog marker is what I'm planning on doing for the timestamped based snapshot, which is going to be far safer than doing an HLog roll and provide less latency. With the roll, you need to not take any writes to the memstore between the roll and the end of the snapshot (otherwise you will lose edits). Doing meta edits into the HLog allows you to keep edits and not worry about it. admin initiated snapshot and admin initiated restore operations as opposed to acting like a read only table. (not sure what happens to "newer" data after a restore, need to reread to see if it is in there, not sure about the cost to restore a snapshot) Yup, right now its all handled from HBaseAdmin. Matteo was interested in working on the restore stuff, but depending on timing, I may end up picking up that work when I get the taking of a snapshot working. I think part of "snapshots" definitely includes getting back the state. I believe it also has an ability to read files directly from an MR job without having to go through HBase's get/put interface. Is that in scope for HBASE-6055 ? Absolutely in scope. It just didn't come up because I considered that part of the restore (which Matteo expressed interest). If you had to go through the high-level interface, then you would just use the procedure Lars talks about in his blog: http://hadoop-hbase.blogspot.com/2012/04/timestamp-consistent-backups-in-hbase.html The other notable change is that I'm building to support multiple snapshots concurrently. Its really a trivial change, so I don't think its too much feature creep, just a matter of using lists rather than a single item. How does this buy your more consistency? Aren't we still inconsistent at the prepare point now instead? Can we just write the special snapshotting hlog entry at initiation of prepare, allowing writes to continue, then adding data elsewhere (META) to mark success in commit? We could then have some compaction/flush time logic cleanup failed atttempt markers? See the above comment about timestamp based vs. point in time and the former being all that's necessary for HBase. This means we don't take downtime and end up with a 'fuzzy' snapshot in terms of global consistency, but is exact in terms of HBase delivered timestamps. The problem point-in-time snapshots overcomes is reaching distributed consensus while still trying to maintain availability and the ability to cross partitions. Since no one has figured out CAP and we are looking for consistency, we have to remove some availability to reach consensus. In this case, the agreement is over the state of the entire table , rather than per region server. Yes, this is strictly against the contract that we have on a Scan, but it is also in line with expectations people have on what a snapshot means. Any writes that are pending before the snapshot are allowed to commit, but any writes that reach the RS after the snapshot time cannot be included in the snapshot. I got a little overzealous in my reading of HBASE-50 and took it to mean global state, but after review the only way it would work within the constraints (no downtime) is to make it timestamp based. But why can't we get global consistency without taking downtime? Let's take your example of using an HLog edit to mark the start (and for ease, lets say the end as well - as long as its durable and recoverable, it doesn't matter if its WAL or META). Say we start a snapshot and send a message to all the RS (lets ignore ZK for the moment, to simplify things) that they should take a snapshot. So they write a marker into the HLog marking the start, create references as mentioned above, and then report to the master that they are done. When everyone is done, we then message each RS to commit the snapshot, which is just another entry into the WAL. Then in rebuilding the snapshot, they would just replay the WAL up to the start (assuming the end is found). How do we know though which writes arrived first on each RS if we just dump a write into the WAL? Ok, so then we need to wait for the MVCC read number to roll forward to when we got the snapshot notification before we can write an edit to the log - totally reasonable. However, the problem arises in attempting to get a global state of the table in a high-volume write environment. We have no guarantee that the "snapshot commit" notification reached each of the RS at the same time. And even if it did reach them at the same time, maybe there was some latency in getting the write number. Or the switch was a little wonky, or it just finishing up a GC (I could go on). Then we have a case where we don't actually have the snapshot as of the commit, but rather "at commit, plus or minus a bit" - not a clean snapshot (if we don't care about being exact then we can do a much faster, lower potential latency solution, the discussion of which is still coming, I promise). In a system that can take millions of writes a second, that is still a non-trivial amount of data that can change in a few milliseconds, no longer a true 'point in time'. The only way to get that global, consistent view is to remove the availability of the table for a short time so we know that the state is the same across all tables. Say we start a snapshot and the start indication doesn't reach the servers and get started at exactly the same time on all the servers , which, as explained above, is very likely. Then we let the servers commit any outstanding writes,but they don't get to take any new writes or a short time. In this time while they are waiting for writes to commit, we can then do all the snapshot preparation (referencing, table info copying). Once we are ready for the snapshot, we report back to the master and wait for the commit step. In this time we are still not taking writes. The key here is that for that short time, none of the servers are taking writes and that allows us to get a single point in time that no writes are committing (but they do get buffered on the server, they just can't change the system state). If we let writes commit, then how do we reach a state that we can agree on across all the servers? If you let the writes commit, you again don't have any assurances that the prepare or the commit message time is agreed to by all the servers. The table-level consistent state is somewhere between the prepare and commit, but it's not clear how one would find that point - I'm pretty sure we can't do this unless we have perfectly synchronized clocks, which is not really possible without a better understanding of quantum mechanics Block writes is a perhaps a bad phrase in this situation. In the current implementation, it buffers the writes as threads into the server, blocking on the updateLock. However, we can go with a "semi-blocking" version: writes still complete, but they aren't going to be visible until we roll forward to the snapshot MVCC number. This lets the writers complete (not affecting latency), but is going to affect read-modify-write and reader-to-writer comparison latency. However, as soon as we roll forward the MVCC, all those writes become visible, essentially catching back up to the current state. A slight modification to the WAL edits will need to be made to write the MVCC number so we can keep track of which writes are in/out of a snapshot, but that shouldn't be too hard (famous last words). You don't even need to modify all the WAL edits, just those made during the snapshot window, so the over the wire cost is still kept essentially the same, when amortized over the life of a table (for the standard use case). I'm looking at doing this once I get the simple version working - one step at a time. Moving to the timestamp based approach lets us keep taking writes but does so at the cost of global consistency in favor of local consistency and still uses the exact same infrastructure . The first patch I'll actually put on RB will be the timestamp based, but let me get the stop the world version going before going down a rabbit hole. The only thing we don't capture is if a writer makes a request to the RS before the snapshot is taken (by another client), but the write doesn't reach the server until after the RS hits the start barrier. From the global client perspective, this write should be in the snapshot, but that requires a single client or client-side write coordination (via a timestamp oracle). However, this is even worse coordination and creates even more constraints on the system where we currently have no coordination between clients (and I'm against adding any). So yes, we miss that edit, but that would be the case in a single-server database anyways without an external timestamp manager (to again distributed coordination between the client and server, though it can be done in a non-blocking manner). I'll mention some of this external coordination in the timestamp explanation.
          Hide
          Jonathan Hsieh added a comment -

          Jesse,

          Thanks for answering the questions. A strong +1 for doing the simplest hbase timestamp-based approach first, and then looking into the more complicated version as an option afterwards. Maybe start a sub issue with the point-in-time approach to move discussion there? (I still have questions there, might be better to ask there)

          The main use case I care about is ability to quickly "snapshot" without downtime and quickly recover it (ideally with no downtime, but possibly with a short downtime window). Although it is a "sloppy snapshot" conceptually it is pretty simple to define and I think the caveats are fairly well undestood. I don't expect something with stronger consistency guarantees than what hbase currently offers but do expect something better (cheaper/faster) than the current closest thing which is a CopyTable.

          I have a bunch of new questions - some just asking for precision and some for clarification. It might be helpful to define terms in the beginning of the doc so it stays consistent?

          • Hm.. how do you restore a snapshot from references files if it hasn't been scan/copied yet? Require scan/copy "materialization" of the snapshot first? (which means slower restore, but probably would likely be simplest for a first cut)
          • Snapshot restore needs to be "transactional" like snapshotting right?
          • what is "export"? is this taking a snapshot or the materialization or the snapshot restore or something else?
          • If we restore snapshots to the same hbase instance, in dir structure, you probably need .regioninfo files as well. (contains region startkey/endkey info necessary to reconsistute META later).
          • Is restoring to a separate instance in scope? If so bulk loads can be expensive – if regions don't line up there will be a bunch of spliting that happens. Again, keeping the regionsinfos and the snapshot's splits may be worthwhile.
          • Where do the materialized versions of the snapshot reference files end up? in the snapshot dirs? elsewhere?
            • This potentially gets a little trickier with markers as opposed to log rolls.
            • The HLog will have edits from regions not relevant to the table's regions. Not a huge problem but maybe an optmization would be that the materialization step will do an "offline hlogsplit/flush" to just keep the data relevent to this table/region?
          Show
          Jonathan Hsieh added a comment - Jesse, Thanks for answering the questions. A strong +1 for doing the simplest hbase timestamp-based approach first, and then looking into the more complicated version as an option afterwards. Maybe start a sub issue with the point-in-time approach to move discussion there? (I still have questions there, might be better to ask there) The main use case I care about is ability to quickly "snapshot" without downtime and quickly recover it (ideally with no downtime, but possibly with a short downtime window). Although it is a "sloppy snapshot" conceptually it is pretty simple to define and I think the caveats are fairly well undestood. I don't expect something with stronger consistency guarantees than what hbase currently offers but do expect something better (cheaper/faster) than the current closest thing which is a CopyTable. I have a bunch of new questions - some just asking for precision and some for clarification. It might be helpful to define terms in the beginning of the doc so it stays consistent? Hm.. how do you restore a snapshot from references files if it hasn't been scan/copied yet? Require scan/copy "materialization" of the snapshot first? (which means slower restore, but probably would likely be simplest for a first cut) Snapshot restore needs to be "transactional" like snapshotting right? what is "export"? is this taking a snapshot or the materialization or the snapshot restore or something else? If we restore snapshots to the same hbase instance, in dir structure, you probably need .regioninfo files as well. (contains region startkey/endkey info necessary to reconsistute META later). Is restoring to a separate instance in scope? If so bulk loads can be expensive – if regions don't line up there will be a bunch of spliting that happens. Again, keeping the regionsinfos and the snapshot's splits may be worthwhile. Where do the materialized versions of the snapshot reference files end up? in the snapshot dirs? elsewhere? This potentially gets a little trickier with markers as opposed to log rolls. The HLog will have edits from regions not relevant to the table's regions. Not a huge problem but maybe an optmization would be that the materialization step will do an "offline hlogsplit/flush" to just keep the data relevent to this table/region?
          Hide
          Jonathan Hsieh added a comment -

          One other place where bulk import can be expensive – if we bulk import all into a single region, it would likely incur a compaction/split storm...

          Show
          Jonathan Hsieh added a comment - One other place where bulk import can be expensive – if we bulk import all into a single region, it would likely incur a compaction/split storm...
          Hide
          Lars Hofhansl added a comment -

          Now that I started to get a bit more familiar with HDFS I am wondering whether HDFS hardlinks (HDFS-3370) or even HDFS snapshots (HDFS-233, HDFS-2802) are not a better avenue. We are looking for data consistency here, which would be better tackled at the data layer.
          Now... Both features are some ways off in HDFS (although we can probably push these forward), so doing something in HBase first is probably needed, but IMHO it should be something quick.
          Lastly if we are considering this for backups HBASE-5547 should be a better (simpler) solution.

          Not trying to derail anything here, just making sure we do not invest a lot of time in vain.

          Show
          Lars Hofhansl added a comment - Now that I started to get a bit more familiar with HDFS I am wondering whether HDFS hardlinks ( HDFS-3370 ) or even HDFS snapshots ( HDFS-233 , HDFS-2802 ) are not a better avenue. We are looking for data consistency here, which would be better tackled at the data layer. Now... Both features are some ways off in HDFS (although we can probably push these forward), so doing something in HBase first is probably needed, but IMHO it should be something quick. Lastly if we are considering this for backups HBASE-5547 should be a better (simpler) solution. Not trying to derail anything here, just making sure we do not invest a lot of time in vain.
          Hide
          Jonathan Hsieh added a comment -

          I agreee that HDFS-3370, and HDFS-233 are likely a ways off. Hardlinks would probably obviate the need for HBASE-5547. We could probably take advantage of HDFS symlinks HDFS-245 which is in Hadoop 2.x.x hdfs but may not be in Hadoop 1.x.x hdfs.

          I think that HBASE-5547 is a prereq for either consistency approach (even if we use symlinks) until we have hdfs hardlinks. I'll take a closer look into HBASE-5547.

          Show
          Jonathan Hsieh added a comment - I agreee that HDFS-3370 , and HDFS-233 are likely a ways off. Hardlinks would probably obviate the need for HBASE-5547 . We could probably take advantage of HDFS symlinks HDFS-245 which is in Hadoop 2.x.x hdfs but may not be in Hadoop 1.x.x hdfs. I think that HBASE-5547 is a prereq for either consistency approach (even if we use symlinks) until we have hdfs hardlinks. I'll take a closer look into HBASE-5547 .
          Hide
          Jesse Yates added a comment -

          @Lars even with the HDFS patches, we should have a way to logically group/backup/restore snapshots.

          IMO hardlinks with HBase snapshots is the way to go, since we can do it with zero downtime, whereas HDFS snapshots require some (though admittedly small) downtime. HBSE-5547 is basically just a hack around hardlinks.

          Show
          Jesse Yates added a comment - @Lars even with the HDFS patches, we should have a way to logically group/backup/restore snapshots. IMO hardlinks with HBase snapshots is the way to go, since we can do it with zero downtime, whereas HDFS snapshots require some (though admittedly small) downtime. HBSE-5547 is basically just a hack around hardlinks.
          Hide
          Lars Hofhansl added a comment -

          IMO hardlinks with HBase snapshots is the way to go... HBSE-5547 is basically just a hack around hardlinks

          Yep.

          Show
          Lars Hofhansl added a comment - IMO hardlinks with HBase snapshots is the way to go... HBSE-5547 is basically just a hack around hardlinks Yep.
          Hide
          Jesse Yates added a comment -

          A couple of definitions going forward:

          • materialization: the end result of taking a single snapshot, on the same cluster. It ends up in in the .snapshot/[snapshot_name] directory
          • export: sending the snapshot to another cluster or another part of the same cluster
          • restore: taking an exported snapshot and converting the snapshot into an active table.

          Hm.. how do you restore a snapshot from references files if it hasn't been scan/copied yet? Require scan/copy "materialization" of the snapshot first? (which means slower restore, but probably would likely be simplest for a first cut)

          Right now, you would do a M/R job to distcp over the files to another cluster or a backup part of your cluster. Since we are just storing references, the actual file copying will be necessary. This will be helped by using the actual "Reference" class for the HFiles (and currently being (mis)used for the WALs, but I don't think we actually need to keep the WALs - I'll comment in the timestamp ticket). Since they are just reference files, you could just use the regular HFile reader to load them into another table.

          Snapshot restore needs to be "transactional" like snapshotting right?

          Yeah, I guess. I don't really see this as a problem - just keep it to one restore at a time. But it would be all or nothing to get a table online.

          what is "export"? is this taking a snapshot or the materialization or the snapshot restore or something else?

          Export is taking a snapshot from the .snapshot/ directory and possibly having a special snapshot distcp to somewhere. I would consider materialization as taking the exported snapshot and then 'hooking it back up' to another cluster (or the same) as a new table. You could throw materialization of the exported snapshot, but they are in fact distinct.

          If we restore snapshots to the same hbase instance, in dir structure, you probably need .regioninfo files as well. (contains region startkey/endkey info necessary to reconsistute META later).

          +1 I'll make sure that gets in

          Is restoring to a separate instance in scope? If so bulk loads can be expensive – if regions don't line up there will be a bunch of spliting that happens. Again, keeping the regionsinfos and the snapshot's splits may be worthwhile.

          I'd say restore is part of this. Should be solved by having the region info. -1 for split/compact storms.

          Where do the materialized versions of the snapshot reference files end up? in the snapshot dirs? elsewhere?

          What do you mean materialized? After taking snapshot, where do the snapshot files end up? In the .snapshot directory. See my earlier comments on the structure.

          This potentially gets a little trickier with markers as opposed to log rolls.

          If we do a log roll, its probably going to take a bit longer. Also, its not going to be applicable to the timestamp approach, since log rolling will necessitate doing some kind of locking, which we should avoid, where the markers will be much faster.

          The HLog will have edits from regions not relevant to the table's regions. Not a huge problem but maybe an optmization would be that the materialization step will do an "offline hlogsplit/flush" to just keep the data relevent to this table/region?

          +1, assuming we need the HLogs. I think there is a minimally impactful way to avoid this altogether.

          Show
          Jesse Yates added a comment - A couple of definitions going forward: materialization: the end result of taking a single snapshot, on the same cluster. It ends up in in the .snapshot/ [snapshot_name] directory export: sending the snapshot to another cluster or another part of the same cluster restore: taking an exported snapshot and converting the snapshot into an active table. Hm.. how do you restore a snapshot from references files if it hasn't been scan/copied yet? Require scan/copy "materialization" of the snapshot first? (which means slower restore, but probably would likely be simplest for a first cut) Right now, you would do a M/R job to distcp over the files to another cluster or a backup part of your cluster. Since we are just storing references, the actual file copying will be necessary. This will be helped by using the actual "Reference" class for the HFiles (and currently being (mis)used for the WALs, but I don't think we actually need to keep the WALs - I'll comment in the timestamp ticket). Since they are just reference files, you could just use the regular HFile reader to load them into another table. Snapshot restore needs to be "transactional" like snapshotting right? Yeah, I guess. I don't really see this as a problem - just keep it to one restore at a time. But it would be all or nothing to get a table online. what is "export"? is this taking a snapshot or the materialization or the snapshot restore or something else? Export is taking a snapshot from the .snapshot/ directory and possibly having a special snapshot distcp to somewhere. I would consider materialization as taking the exported snapshot and then 'hooking it back up' to another cluster (or the same) as a new table. You could throw materialization of the exported snapshot, but they are in fact distinct. If we restore snapshots to the same hbase instance, in dir structure, you probably need .regioninfo files as well. (contains region startkey/endkey info necessary to reconsistute META later). +1 I'll make sure that gets in Is restoring to a separate instance in scope? If so bulk loads can be expensive – if regions don't line up there will be a bunch of spliting that happens. Again, keeping the regionsinfos and the snapshot's splits may be worthwhile. I'd say restore is part of this. Should be solved by having the region info. -1 for split/compact storms. Where do the materialized versions of the snapshot reference files end up? in the snapshot dirs? elsewhere? What do you mean materialized? After taking snapshot, where do the snapshot files end up? In the .snapshot directory. See my earlier comments on the structure. This potentially gets a little trickier with markers as opposed to log rolls. If we do a log roll, its probably going to take a bit longer. Also, its not going to be applicable to the timestamp approach, since log rolling will necessitate doing some kind of locking, which we should avoid, where the markers will be much faster. The HLog will have edits from regions not relevant to the table's regions. Not a huge problem but maybe an optmization would be that the materialization step will do an "offline hlogsplit/flush" to just keep the data relevent to this table/region? +1, assuming we need the HLogs. I think there is a minimally impactful way to avoid this altogether.
          Hide
          Ted Yu added a comment -

          The HLog will have edits from regions not relevant to the table's regions.

          Over in HBASE-5699, each one of the multiple WALs can be devised to receive edits from single table.

          Show
          Ted Yu added a comment - The HLog will have edits from regions not relevant to the table's regions. Over in HBASE-5699 , each one of the multiple WALs can be devised to receive edits from single table.
          Hide
          gaojinchao added a comment -

          Hi Jesse
          I am considering the solution which don't use Hlog. The way is only handling the memstore and asynchronous flush the memstore to Hfile. when the region server is down, we can finish flushing Hfile by replay editLog. Do you think whether it is feasible or not?
          If we can do, there are several relatively large benefits:
          1. restore the snapshot is easier
          2. We can achieve an incremental backup by HFile

          Show
          gaojinchao added a comment - Hi Jesse I am considering the solution which don't use Hlog. The way is only handling the memstore and asynchronous flush the memstore to Hfile. when the region server is down, we can finish flushing Hfile by replay editLog. Do you think whether it is feasible or not? If we can do, there are several relatively large benefits: 1. restore the snapshot is easier 2. We can achieve an incremental backup by HFile
          Hide
          Jesse Yates added a comment -

          gaojinchao In the "stop the world" impementation, flushing the HFile is going to take too long. However, in the timestamp based approach time doesn't play as big a role (oh the irony!), so we can actually flush the HFiles and do what you are talking about. I'm most of the way through a writeup for how this would work, but have been a bit busy the last few days to post it - planning to have it up tomorrow in a sub-ticket (as Jon suggests).

          Show
          Jesse Yates added a comment - gaojinchao In the "stop the world" impementation, flushing the HFile is going to take too long. However, in the timestamp based approach time doesn't play as big a role (oh the irony!), so we can actually flush the HFiles and do what you are talking about. I'm most of the way through a writeup for how this would work, but have been a bit busy the last few days to post it - planning to have it up tomorrow in a sub-ticket (as Jon suggests).
          Hide
          gaojinchao added a comment -

          Fine, Thanks, I will take some time for this feature.

          Show
          gaojinchao added a comment - Fine, Thanks, I will take some time for this feature.
          Hide
          Jonathan Hsieh added a comment -

          I still a bit confused – still at the basic admin level. I think it would help if we give the "restoring"/"export" parts some more attention and talk about usage as opposed to mechanism first. I'm going to pose some use case/examples/scenarios which hopefully will be easier to discuss.

          Let's say I am an admin, and we are pre hdfs hardlinks.

          I issue a "snapshot" command at the shell/master.

          • HBase creates a new .snapshot subdir, and it contains references to HLogs and HFiles. This is a "snapshot"
            • This step is called: snapshotting, "taking a snapshot", and also materializing right?

          I currently have a snapshot. I want read-only access its contents to compare with the current table.

          • Does HBase know how to interpret the stuff in a .snapshot dir such that it act like a read-only table?
          • Do I, as an admin, need to execute some step to make it appear in HBase as a read-only table? (if so what is this called?)

          I currently have a snapshot. Oops! I accidentally truncated the table I had snapshotted. I don't want the truncated version of the table anymore and I want to replace the table with the snapshot so I have read write access.

          • This is called "restoring" the snapshot right? (and I do this by issuing a something like "restore" command at the shell?)
          • Does HBase copy or move the data referred to in the snapshot?

          I currently have a snapshot. I want the current version but I'd like to clone of the snapshotted table that provides read/write access to the clone.

          • Is/should this be supported?
          • Is this called "restoring" or "exporting" the snapshot (to a new name)?
          • For this to work I need to convert all references into actual copies of the HFiles and HLogs right? Is this conversion called exporting? (FYI, this is what I meant materializing to mean, but let's just stick to your definitions)

          I currently have a snapshot. I want to send a copy of the snapshot to a remote cluster so that it can provide read/write access to the data.

          • Is/should this be supported?
          • Do both HBase instances need to be up at the same time?
            • This process would need to dereference the snapshot's references and copy them. What is it called? exporting?

          Source of confusion

          Export is taking a snapshot from the .snapshot/ directory and possibly having a special snapshot distcp to somewhere. I would consider materialization as taking the exported snapshot and then 'hooking it back up' to another cluster (or the same) as a new table. You could throw materialization of the exported snapshot, but they are in fact distinct.

          I think the first "materialization" is supposed to be "restoration" yeah? I don't quite get the last sentence.

          Show
          Jonathan Hsieh added a comment - I still a bit confused – still at the basic admin level. I think it would help if we give the "restoring"/"export" parts some more attention and talk about usage as opposed to mechanism first. I'm going to pose some use case/examples/scenarios which hopefully will be easier to discuss. Let's say I am an admin, and we are pre hdfs hardlinks. I issue a "snapshot" command at the shell/master. HBase creates a new .snapshot subdir, and it contains references to HLogs and HFiles. This is a "snapshot" This step is called: snapshotting, "taking a snapshot", and also materializing right? I currently have a snapshot. I want read-only access its contents to compare with the current table. Does HBase know how to interpret the stuff in a .snapshot dir such that it act like a read-only table? Do I, as an admin, need to execute some step to make it appear in HBase as a read-only table? (if so what is this called?) I currently have a snapshot. Oops! I accidentally truncated the table I had snapshotted. I don't want the truncated version of the table anymore and I want to replace the table with the snapshot so I have read write access. This is called "restoring" the snapshot right? (and I do this by issuing a something like "restore" command at the shell?) Does HBase copy or move the data referred to in the snapshot? I currently have a snapshot. I want the current version but I'd like to clone of the snapshotted table that provides read/write access to the clone. Is/should this be supported? Is this called "restoring" or "exporting" the snapshot (to a new name)? For this to work I need to convert all references into actual copies of the HFiles and HLogs right? Is this conversion called exporting? (FYI, this is what I meant materializing to mean, but let's just stick to your definitions) I currently have a snapshot. I want to send a copy of the snapshot to a remote cluster so that it can provide read/write access to the data. Is/should this be supported? Do both HBase instances need to be up at the same time? This process would need to dereference the snapshot's references and copy them. What is it called? exporting? Source of confusion Export is taking a snapshot from the .snapshot/ directory and possibly having a special snapshot distcp to somewhere. I would consider materialization as taking the exported snapshot and then 'hooking it back up' to another cluster (or the same) as a new table. You could throw materialization of the exported snapshot, but they are in fact distinct. I think the first "materialization" is supposed to be "restoration" yeah? I don't quite get the last sentence.
          Hide
          Matteo Bertozzi added a comment -

          @Jon inline replies

          I issue a "snapshot" command at the shell/master.

          • HBase creates a new .snapshot subdir, and it contains references to HLogs and HFiles. This is a "snapshot"
            • This step is called: snapshotting, "taking a snapshot", and also materializing right?

          Yes, When you issue a "snapshot" command, hbase create a new .snapshot subdir containing references to hlog and hfile.
          This is "taking a snapshot" or snapshotting...
          but not materialization, I think "materialization" is when you copy the hfiles/hlogs somewhere else...

          I currently have a snapshot. I want read-only access its contents to compare with the current table.

          • Does HBase know how to interpret the stuff in a .snapshot dir such that it act like a read-only table?
          • Do I, as an admin, need to execute some step to make it appear in HBase as a read-only table? (if so what is this called?)

          I think that the first point is more like a snapshot-scan... that scan the hfiles + hlog in the snapshot directory and show you the result...
          The second point seems more like a "Restore on different table" and marking the table as readonly

          I currently have a snapshot. Oops! I accidentally truncated the table I had snapshotted. I don't want the truncated version of the table anymore and I want to replace the table with the snapshot so I have read write access.

          • This is called "restoring" the snapshot right? (and I do this by issuing a something like "restore" command at the shell?)
          • Does HBase copy or move the data referred to in the snapshot?

          "Restore" is when you replace your current table with the snapshot version, and you do it by "restore snapshot-name"
          Yeah you need to copy the "old hfiles" to restore the snapshot (but maybe not every hfiles are removed from the current table)

          I currently have a snapshot. I want the current version but I'd like to clone of the snapshotted table that provides read/write access to the clone.

          • Is/should this be supported?
          • Is this called "restoring" or "exporting" the snapshot (to a new name)?
          • For this to work I need to convert all references into actual copies of the HFiles and HLogs right? Is this conversion called exporting? (FYI, this is what I meant materializing to mean, but let's just stick to your definitions)

          Yeah this is really easy with HardLink... some more work is needed to keep track of references files
          This is "Restore on a different table", "export" is when you're copying the .snapshot/name folder to another cluster...
          If you think in term of HardLink you don't need to copy the hfiles but just doing an HardLink... more code is needed to use Reference Files but you can avoid the copy. (Note that HLog need to be replayed, so this is the only one that need to be copied.

          I currently have a snapshot. I want to send a copy of the snapshot to a remote cluster so that it can provide read/write access to the data.
          Is/should this be supported?

          • Do both HBase instances need to be up at the same time?
          • This process would need to dereference the snapshot's references and copy them. What is it called? exporting?

          Yes this is "Import/Export" that besically is a distcp of the .snapshot/name folder
          I Think that is enough having both hdfs up at the same time.
          Yeah in this case you need to physically copy the hfiles.

          Show
          Matteo Bertozzi added a comment - @Jon inline replies I issue a "snapshot" command at the shell/master. HBase creates a new .snapshot subdir, and it contains references to HLogs and HFiles. This is a "snapshot" This step is called: snapshotting, "taking a snapshot", and also materializing right? Yes, When you issue a "snapshot" command, hbase create a new .snapshot subdir containing references to hlog and hfile. This is "taking a snapshot" or snapshotting... but not materialization, I think "materialization" is when you copy the hfiles/hlogs somewhere else... I currently have a snapshot. I want read-only access its contents to compare with the current table. Does HBase know how to interpret the stuff in a .snapshot dir such that it act like a read-only table? Do I, as an admin, need to execute some step to make it appear in HBase as a read-only table? (if so what is this called?) I think that the first point is more like a snapshot-scan... that scan the hfiles + hlog in the snapshot directory and show you the result... The second point seems more like a "Restore on different table" and marking the table as readonly I currently have a snapshot. Oops! I accidentally truncated the table I had snapshotted. I don't want the truncated version of the table anymore and I want to replace the table with the snapshot so I have read write access. This is called "restoring" the snapshot right? (and I do this by issuing a something like "restore" command at the shell?) Does HBase copy or move the data referred to in the snapshot? "Restore" is when you replace your current table with the snapshot version, and you do it by "restore snapshot-name" Yeah you need to copy the "old hfiles" to restore the snapshot (but maybe not every hfiles are removed from the current table) I currently have a snapshot. I want the current version but I'd like to clone of the snapshotted table that provides read/write access to the clone. Is/should this be supported? Is this called "restoring" or "exporting" the snapshot (to a new name)? For this to work I need to convert all references into actual copies of the HFiles and HLogs right? Is this conversion called exporting? (FYI, this is what I meant materializing to mean, but let's just stick to your definitions) Yeah this is really easy with HardLink... some more work is needed to keep track of references files This is "Restore on a different table", "export" is when you're copying the .snapshot/name folder to another cluster... If you think in term of HardLink you don't need to copy the hfiles but just doing an HardLink... more code is needed to use Reference Files but you can avoid the copy. (Note that HLog need to be replayed, so this is the only one that need to be copied. I currently have a snapshot. I want to send a copy of the snapshot to a remote cluster so that it can provide read/write access to the data. Is/should this be supported? Do both HBase instances need to be up at the same time? This process would need to dereference the snapshot's references and copy them. What is it called? exporting? Yes this is "Import/Export" that besically is a distcp of the .snapshot/name folder I Think that is enough having both hdfs up at the same time. Yeah in this case you need to physically copy the hfiles.
          Hide
          Lars Hofhansl added a comment -

          Let's try to avoid going overboard here.
          In principle snapshot and backup/restore are different and independent.

          A snapshot generates a consistent snapshot of the data that can subsequently be copied conveniently somewhere else - thus creating a backup.

          Ideally we would not even prescribe the backup/restore semantics here, but just provide missing building blocks.

          Just my $0.02.

          Another thought here is: In principle an HFile resulting from a major compaction could be considered a baseline copy and additional HFiles would be incremental changes on top of that baseline. It might be worth considering if we can make use of this ability of HBase to overlay changes from many sources into a single view of the data (would probably be tricky as regions are flushed in sync, etc, just waving hands here).

          Show
          Lars Hofhansl added a comment - Let's try to avoid going overboard here. In principle snapshot and backup/restore are different and independent. A snapshot generates a consistent snapshot of the data that can subsequently be copied conveniently somewhere else - thus creating a backup. Ideally we would not even prescribe the backup/restore semantics here, but just provide missing building blocks. Just my $0.02. Another thought here is: In principle an HFile resulting from a major compaction could be considered a baseline copy and additional HFiles would be incremental changes on top of that baseline. It might be worth considering if we can make use of this ability of HBase to overlay changes from many sources into a single view of the data (would probably be tricky as regions are flushed in sync, etc, just waving hands here).
          Hide
          Jonathan Hsieh added a comment -

          Hey Lars,

          Sorry if it seems like I'm going overboard – I'm trying to tease out consistent common definitions, and get an explicit high-level understanding of how the feature is supposed to be used from an user/admin point of view.

          I'm also trying to understand what is in scope and not (ex: making the snapshot act like a read-only table could be in scope, restoring/replacing the original table should be, but restoring to another name could be deferred until hardlinks, the import/export stuff could be done using existing means.)

          Show
          Jonathan Hsieh added a comment - Hey Lars, Sorry if it seems like I'm going overboard – I'm trying to tease out consistent common definitions, and get an explicit high-level understanding of how the feature is supposed to be used from an user/admin point of view. I'm also trying to understand what is in scope and not (ex: making the snapshot act like a read-only table could be in scope, restoring/replacing the original table should be, but restoring to another name could be deferred until hardlinks, the import/export stuff could be done using existing means.)
          Hide
          Jesse Yates added a comment -

          Adding link to HBASE-6230 for further discussion around snapshot restore.

          Show
          Jesse Yates added a comment - Adding link to HBASE-6230 for further discussion around snapshot restore.
          Hide
          Jesse Yates added a comment -

          Pushed up initial review (based on HBASE-5547) on review board: https://reviews.apache.org/r/5817/

          Show
          Jesse Yates added a comment - Pushed up initial review (based on HBASE-5547 ) on review board: https://reviews.apache.org/r/5817/
          Hide
          Ted Yu added a comment -

          Can you rebase to current trunk ?

          |--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          |+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          --------------------------
          File to patch: ^C
          
          Show
          Ted Yu added a comment - Can you rebase to current trunk ? |--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java |+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java -------------------------- File to patch: ^C
          Hide
          Jesse Yates added a comment -

          Looks like I'm just missing HBASE-6341 and HBASE-6283, both of which don't really change the patch. Keep in mind you will need to apply the latest patch from https://reviews.apache.org/r/4633/ (RB of latest for HBASE-5547, that code on trunk) before applying the patch - RB isn't obvious about having a parent diff.

          Unless something has been committed to the svn that is significantly different and hasn't propagated to the git repo yet...

          Show
          Jesse Yates added a comment - Looks like I'm just missing HBASE-6341 and HBASE-6283 , both of which don't really change the patch. Keep in mind you will need to apply the latest patch from https://reviews.apache.org/r/4633/ (RB of latest for HBASE-5547 , that code on trunk) before applying the patch - RB isn't obvious about having a parent diff. Unless something has been committed to the svn that is significantly different and hasn't propagated to the git repo yet...
          Hide
          Matteo Bertozzi added a comment -

          @Jesse working on HBASE-6353, I've also switched to use protobuf (HMasterInterface was removed HBASE-6039).
          Maybe you can use when rebasing on trunk (HBaseAdmin, MasterAdminProtocol, ...)

          Show
          Matteo Bertozzi added a comment - @Jesse working on HBASE-6353 , I've also switched to use protobuf (HMasterInterface was removed HBASE-6039 ). Maybe you can use when rebasing on trunk (HBaseAdmin, MasterAdminProtocol, ...)
          Hide
          Jesse Yates added a comment -

          @Matteo I've done that already, looks like my diff-ing got messed up :/ Working on pushing up a new patch...

          Show
          Jesse Yates added a comment - @Matteo I've done that already, looks like my diff-ing got messed up :/ Working on pushing up a new patch...
          Hide
          Ted Yu added a comment -

          Flipping through 5 pages on review board is slow. So I am putting down some notes here.

          For HStore.java:
          The license header doesn't look like the standard format.
          Please add audience and stability annotations to this new interface.

          +  FileStatus[] getStoreFiles() throws IOException;
          +
          +  List<StoreFile> getStorefiles();
          

          Why do we need two methods which are spelled almost the same, yet returning different types ? When refactoring, we should make the code cleaner.
          There're many methods which don't have javadoc. Please add javadoc for them.

          +  public HStore getDelgate() {
          

          Correct spelling for the above method.

          Show
          Ted Yu added a comment - Flipping through 5 pages on review board is slow. So I am putting down some notes here. For HStore.java: The license header doesn't look like the standard format. Please add audience and stability annotations to this new interface. + FileStatus[] getStoreFiles() throws IOException; + + List<StoreFile> getStorefiles(); Why do we need two methods which are spelled almost the same, yet returning different types ? When refactoring, we should make the code cleaner. There're many methods which don't have javadoc. Please add javadoc for them. + public HStore getDelgate() { Correct spelling for the above method.
          Hide
          Jonathan Hsieh added a comment -

          @Ted – FYI, I keep 5 screens open each on a different one. Then I can flip between them quick and comment on each.

          Show
          Jonathan Hsieh added a comment - @Ted – FYI, I keep 5 screens open each on a different one. Then I can flip between them quick and comment on each.
          Hide
          Ted Yu added a comment -

          Thanks for the hint, Jon.
          I thought of that approach.

          I recently looked up related classes in the patch using vi directly.
          It would be nice if we can reduce the number of classes: controller, monitor, manager, sentinel, etc. It is hard to follow

          I have gone through about 2.5 pages of diff.
          I can see there is more work to be done for Global snapshot.

          Show
          Ted Yu added a comment - Thanks for the hint, Jon. I thought of that approach. I recently looked up related classes in the patch using vi directly. It would be nice if we can reduce the number of classes: controller, monitor, manager, sentinel, etc. It is hard to follow I have gone through about 2.5 pages of diff. I can see there is more work to be done for Global snapshot.
          Hide
          Jesse Yates added a comment -

          I've working on a new version that abstracts out a lot of the different pieces in the snapshot that can be reused (both here and in general in HBase) and then reimplementing everything based on those abstractions. Each of these abstractions/standalone pieces is going to moved to another jira and given their own RB review. Hopefully this is going to makes reviewer's lives easier.

          I currently have tests passing and hopefully, I'll have a new version (and child tickets) up on RB tomorrow.

          Show
          Jesse Yates added a comment - I've working on a new version that abstracts out a lot of the different pieces in the snapshot that can be reused (both here and in general in HBase) and then reimplementing everything based on those abstractions. Each of these abstractions/standalone pieces is going to moved to another jira and given their own RB review. Hopefully this is going to makes reviewer's lives easier. I currently have tests passing and hopefully, I'll have a new version (and child tickets) up on RB tomorrow.
          Hide
          Andrew Purtell added a comment -

          Each of these abstractions/standalone pieces is going to moved to another jira and given their own RB review.

          Jesse Yates So maybe we can use this JIRA as an umbrella for subtasks?

          Show
          Andrew Purtell added a comment - Each of these abstractions/standalone pieces is going to moved to another jira and given their own RB review. Jesse Yates So maybe we can use this JIRA as an umbrella for subtasks?
          Hide
          Jesse Yates added a comment -

          Created sub-tasks HBASE-6568, HBASE-6569, HBASE-6570, HBASE-6571, HBASE-6573 for each of the pieces for snapshots. Posting a new snapshots patch (based on all these patches - to apply the coming patch you will need to apply all the others first) to RB soon.

          Show
          Jesse Yates added a comment - Created sub-tasks HBASE-6568 , HBASE-6569 , HBASE-6570 , HBASE-6571 , HBASE-6573 for each of the pieces for snapshots. Posting a new snapshots patch (based on all these patches - to apply the coming patch you will need to apply all the others first) to RB soon.
          Hide
          Jesse Yates added a comment -

          Current patch is up on RB, but is missing (1) validation of the snapshot on the master, (2) concurrent flushes during a timestamp-consistent snapshot and (3) concurrent compactions during a timestamp-consistent snapshot. That being said, these are relatively minor elements that can be rolled in after a majority of reviews.

          I've currently got (2) working with unit tests and am hoping to push up a new version early next week with all 3 elements (so a complete implementation).

          Show
          Jesse Yates added a comment - Current patch is up on RB, but is missing (1) validation of the snapshot on the master, (2) concurrent flushes during a timestamp-consistent snapshot and (3) concurrent compactions during a timestamp-consistent snapshot. That being said, these are relatively minor elements that can be rolled in after a majority of reviews. I've currently got (2) working with unit tests and am hoping to push up a new version early next week with all 3 elements (so a complete implementation).
          Hide
          Jesse Yates added a comment -

          Looking at implementing concurrent compactions, there is an issue around allowing compactions and how to get a consistent view of the directory for each store. If a compaction is taking place and we 'ls' the directory for a store, the following may occur (or some semantically correct subset of the following):

          • get the first set of HFiles in the directory
          • compaction removes all the files
          • compaction moves in its own files
          • we get the next batch of files from the namenode for the original 'ls'

          This leads to a munged (not necessarily incorrect) view of the hfiles that will require another compaction on restore to get a reasonable performance. There are a couple considerations here.

          (1) the above situation occurs only when we have more files in a store that the ls limit on the namenode, which is 1000 by default - the unit of atomicity. As long as a single store doesn't have more than 1000 files, then we can just ignore compactions entirely and snap away. However, once we breach 1000 files, this becomes a different, potentially far more complex to reason about, problem.

          (2) We can block for the currently running compactions to finish and then get a quick 'ls' between compactions starting. This is a bit more intrustive and will potentialy hold up the compaction queue for a little bit. Also as we have more files and a more active system it becomes increasingly likely to get a compaction and cause your snapshot to fail as it waits on the compaction to finish (since we time-bound snapshots to minimize impact on the system).

          Personally, it seems unlikely that we are going to get more than 1000 files in a single store. However, if its unlikely, that means its probably going to happen Option (2) is far more intrusive and code intensive, potentially causing some lag in the system, but is sure to be safe once we get it right.

          Thoughts?

          Show
          Jesse Yates added a comment - Looking at implementing concurrent compactions, there is an issue around allowing compactions and how to get a consistent view of the directory for each store. If a compaction is taking place and we 'ls' the directory for a store, the following may occur (or some semantically correct subset of the following): get the first set of HFiles in the directory compaction removes all the files compaction moves in its own files we get the next batch of files from the namenode for the original 'ls' This leads to a munged (not necessarily incorrect) view of the hfiles that will require another compaction on restore to get a reasonable performance. There are a couple considerations here. (1) the above situation occurs only when we have more files in a store that the ls limit on the namenode , which is 1000 by default - the unit of atomicity. As long as a single store doesn't have more than 1000 files, then we can just ignore compactions entirely and snap away. However, once we breach 1000 files, this becomes a different, potentially far more complex to reason about, problem. (2) We can block for the currently running compactions to finish and then get a quick 'ls' between compactions starting. This is a bit more intrustive and will potentialy hold up the compaction queue for a little bit. Also as we have more files and a more active system it becomes increasingly likely to get a compaction and cause your snapshot to fail as it waits on the compaction to finish (since we time-bound snapshots to minimize impact on the system). Personally, it seems unlikely that we are going to get more than 1000 files in a single store. However, if its unlikely, that means its probably going to happen Option (2) is far more intrusive and code intensive, potentially causing some lag in the system, but is sure to be safe once we get it right. Thoughts?
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3237 (See https://builds.apache.org/job/HBase-TRUNK/3237/)
          HBASE-6055 Fix hfile/log cleaning delegate method naming (Revision 1374478)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3237 (See https://builds.apache.org/job/HBase-TRUNK/3237/ ) HBASE-6055 Fix hfile/log cleaning delegate method naming (Revision 1374478) Result = SUCCESS stack : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
          Hide
          Jonathan Hsieh added a comment -

          Hm.. previous should have been labeled HBASE-6570.

          Show
          Jonathan Hsieh added a comment - Hm.. previous should have been labeled HBASE-6570 .
          Hide
          Matteo Bertozzi added a comment -

          Can we have a snapshot branch to speedup the development?

          currently to test snapshot you need to apply patches from different jiras,
          and after a while they cannot be applied cleanly,
          and it require some time to been able to run it...

          since neither me or Jesse has rights to commit, do you have any idea how to do that? someone volunteer to merge the patches in the branch? any other suggestions?

          Show
          Matteo Bertozzi added a comment - Can we have a snapshot branch to speedup the development? currently to test snapshot you need to apply patches from different jiras, and after a while they cannot be applied cleanly, and it require some time to been able to run it... since neither me or Jesse has rights to commit, do you have any idea how to do that? someone volunteer to merge the patches in the branch? any other suggestions?
          Hide
          Lars Hofhansl added a comment -

          I think that makes sense at this point.

          I've also been thinking that a synchronous flush of all regions of a table (or cluster) would already get us far along a backup solution (with HBASE-5547). So maybe we should start with that and do the rest in a branch.

          Show
          Lars Hofhansl added a comment - I think that makes sense at this point. I've also been thinking that a synchronous flush of all regions of a table (or cluster) would already get us far along a backup solution (with HBASE-5547 ). So maybe we should start with that and do the rest in a branch.
          Hide
          Jesse Yates added a comment -

          Just setup my github repo for a snapshots development branch: https://github.com/jyates/hbase/tree/snapshots

          We can make it such that any of the future patches for snapshots (HBASE-6765, HBASE-6353, HBASE-6571, HBASE-6573) all go into this branch and then we just merge the branch into svn with 3 +1's from committers when its ready (as per the discussion here: http://search-hadoop.com/m/asM982C5FkS1/hbase+branch+git&subj=Thoughts+about+large+feature+dev+branches).

          All reviews still go through reviewboard and will receive the same scrutiny, but get committed over on github until we want to roll it into trunk.

          Thoughts?

          Show
          Jesse Yates added a comment - Just setup my github repo for a snapshots development branch: https://github.com/jyates/hbase/tree/snapshots We can make it such that any of the future patches for snapshots ( HBASE-6765 , HBASE-6353 , HBASE-6571 , HBASE-6573 ) all go into this branch and then we just merge the branch into svn with 3 +1's from committers when its ready (as per the discussion here: http://search-hadoop.com/m/asM982C5FkS1/hbase+branch+git&subj=Thoughts+about+large+feature+dev+branches ). All reviews still go through reviewboard and will receive the same scrutiny, but get committed over on github until we want to roll it into trunk. Thoughts?
          Hide
          Jonathan Hsieh added a comment -

          +1 Sounds good to me. We might have to do the incremental reviews(generating a parent patch and then the main patch) to send up to review board but this should work.

          Show
          Jonathan Hsieh added a comment - +1 Sounds good to me. We might have to do the incremental reviews(generating a parent patch and then the main patch) to send up to review board but this should work.
          Hide
          Jesse Yates added a comment -

          @Jon yeah, that's the pain of RB, but you can just do 'git checkout HEAD~1; git diff trunk' to generate that parent patch - not too much overhead.

          Show
          Jesse Yates added a comment - @Jon yeah, that's the pain of RB, but you can just do 'git checkout HEAD~1; git diff trunk' to generate that parent patch - not too much overhead.
          Hide
          Jesse Yates added a comment -

          I was going through the offline snapshot code (https://github.com/jyates/hbase/tree/offline-snapshots) and noticed that apparently I wrote the following:

          Path editsdir = HLog.getRegionDirRecoveredEditsDir(HRegion.getRegionDir(tdir,regionInfo.getEncodedName()));
          WALReferenceTask op = new WALReferenceTask(snapshot, this.monitor, editsdir, conf, fs, "disabledTableSnapshot");
          

          For referencing the current hfiles for a disabled table, this makes no sense. However, it got me thinking about dealing with recovered edits for a table. Even if a table is disabled, it may have recovered edits that haven't been applied to the table (a RS comes up, splits the logs, but then dies again before replaying the split log).

          If I'm reading the log-splitting code correctly, I think it archives the original HLog after splitting, but not before the edits are applied to the region. This would mean we also need to reference the recovered.edits directory under each region, if we keep the current implementation...right?

          I was thinking that instead we can keep the hfiles around in the .logs directory until the recovered.edits files for that log file have been replayed. This way we can avoid another task for snapshotting (referencing all the recovered edits) and keep everything simple fairly simple. There would need to be some extra work to keep track of the source hlog - either an 'info' file for the source hlog that lists the written recovered.edits files or special naming of the recovered.edits files that point back to the source file.

          Thoughts?

          Show
          Jesse Yates added a comment - I was going through the offline snapshot code ( https://github.com/jyates/hbase/tree/offline-snapshots ) and noticed that apparently I wrote the following: Path editsdir = HLog.getRegionDirRecoveredEditsDir(HRegion.getRegionDir(tdir,regionInfo.getEncodedName())); WALReferenceTask op = new WALReferenceTask(snapshot, this .monitor, editsdir, conf, fs, "disabledTableSnapshot" ); For referencing the current hfiles for a disabled table, this makes no sense. However, it got me thinking about dealing with recovered edits for a table. Even if a table is disabled, it may have recovered edits that haven't been applied to the table (a RS comes up, splits the logs, but then dies again before replaying the split log). If I'm reading the log-splitting code correctly, I think it archives the original HLog after splitting, but not before the edits are applied to the region. This would mean we also need to reference the recovered.edits directory under each region, if we keep the current implementation...right? I was thinking that instead we can keep the hfiles around in the .logs directory until the recovered.edits files for that log file have been replayed. This way we can avoid another task for snapshotting (referencing all the recovered edits) and keep everything simple fairly simple. There would need to be some extra work to keep track of the source hlog - either an 'info' file for the source hlog that lists the written recovered.edits files or special naming of the recovered.edits files that point back to the source file. Thoughts?
          Hide
          Matteo Bertozzi added a comment -

          When you're talking about hfiles, you are referring to the log files right? I've a bit a of confusion reading your comment, bacause the log files are sequence files. anyway...

          The logs in /hbase/.logs are splitted (new files are created in region/recover.edits) and if you look at HRegion.replayRecoveredEditsIfAny(), the content of recover.edits is removed as soon as the edits are applied. Removed, not archived. And this means that as soon as the table goes online, the snapshot doesn't have a way to read those files.

          but as you've said, the original (full) log is still available during split, but moved to the archive (.oldlogs) as soon as the split is done.

          This means that if you see files in recover.edits, you should have the full logs in /hbase/.logs folder. And you can keep a reference to them, as you do for the online snapshot.

          Another semi-unrelated note... currently we keep full logs files, and the restore needs to split them (see the restore code SnapshotLogSplitter, https://github.com/matteobertozzi/hbase/blob/snapshot-dev/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/restore/RestoreSnapshotHelper.java#L398)
          Can we move this logic at the end of the take snapshot operation and split the logs in .snapshot/region/recover.edits?

          Show
          Matteo Bertozzi added a comment - When you're talking about hfiles, you are referring to the log files right? I've a bit a of confusion reading your comment, bacause the log files are sequence files. anyway... The logs in /hbase/.logs are splitted (new files are created in region/recover.edits) and if you look at HRegion.replayRecoveredEditsIfAny(), the content of recover.edits is removed as soon as the edits are applied. Removed, not archived. And this means that as soon as the table goes online, the snapshot doesn't have a way to read those files. but as you've said, the original (full) log is still available during split, but moved to the archive (.oldlogs) as soon as the split is done. This means that if you see files in recover.edits, you should have the full logs in /hbase/.logs folder. And you can keep a reference to them, as you do for the online snapshot. Another semi-unrelated note... currently we keep full logs files, and the restore needs to split them (see the restore code SnapshotLogSplitter, https://github.com/matteobertozzi/hbase/blob/snapshot-dev/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/restore/RestoreSnapshotHelper.java#L398 ) Can we move this logic at the end of the take snapshot operation and split the logs in .snapshot/region/recover.edits?
          Hide
          Jesse Yates added a comment -

          When you're talking about hfiles, you are referring to the log files right? I've a bit a of confusion reading your comment, bacause the log files are sequence files. anyway...

          Oops, typing tired. Yeah, I mean hlogs the entire time.

          The logs in /hbase/.logs are splitted (new files are created in region/recover.edits) and if you look at HRegion.replayRecoveredEditsIfAny(), the content of recover.edits is removed as soon as the edits are applied. Removed, not archived. And this means that as soon as the table goes online, the snapshot doesn't have a way to read those files.

          but as you've said, the original (full) log is still available during split, but moved to the archive (.oldlogs) as soon as the split is done.

          This means that if you see files in recover.edits, you should have the full logs in /hbase/.logs folder. And you can keep a reference to them, as you do for the online snapshot

          Keeping all the logs in .oldlogs as well as .logs will cover a LOT more hlogs than are necessary to restore the table. Better would be just just reference all the files in the recovered.edits directory, but I worry that there will probably be some race conditions (especially in cases where a server is brought up and down multiple times). Easier just seems to be to remove the log file when when all the recovered.edits are finished. For instance, we could use the FileLink stuff Matteo is working on to ref-count that hlog and only delete it when the last 'reference' (or file derived from that hlog) is gone from the recovered.edits directory

          Show
          Jesse Yates added a comment - When you're talking about hfiles, you are referring to the log files right? I've a bit a of confusion reading your comment, bacause the log files are sequence files. anyway... Oops, typing tired. Yeah, I mean hlogs the entire time. The logs in /hbase/.logs are splitted (new files are created in region/recover.edits) and if you look at HRegion.replayRecoveredEditsIfAny(), the content of recover.edits is removed as soon as the edits are applied. Removed, not archived. And this means that as soon as the table goes online, the snapshot doesn't have a way to read those files. but as you've said, the original (full) log is still available during split, but moved to the archive (.oldlogs) as soon as the split is done. This means that if you see files in recover.edits, you should have the full logs in /hbase/.logs folder. And you can keep a reference to them, as you do for the online snapshot Keeping all the logs in .oldlogs as well as .logs will cover a LOT more hlogs than are necessary to restore the table. Better would be just just reference all the files in the recovered.edits directory, but I worry that there will probably be some race conditions (especially in cases where a server is brought up and down multiple times). Easier just seems to be to remove the log file when when all the recovered.edits are finished. For instance, we could use the FileLink stuff Matteo is working on to ref-count that hlog and only delete it when the last 'reference' (or file derived from that hlog) is gone from the recovered.edits directory
          Hide
          Jesse Yates added a comment -

          Another semi-unrelated note... currently we keep full logs files, and the restore needs to split them (see the restore code SnapshotLogSplitter, https://github.com/matteobertozzi/hbase/blob/snapshot-dev/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/restore/RestoreSnapshotHelper.java#L398)
          Can we move this logic at the end of the take snapshot operation and split the logs in .snapshot/region/recover.edits?

          If we move it into the snapshot operation, then that will slow down the overall operation and make it more difficult to reason about how long a snapshot 'should' take. In particular, this becomes difficult because we want to give the client firm time bounds, but the log splitting is not time bounded (AFAIK).

          An alternative would be to have a background snapshot-log-splitter task that just goes through and splits logs for snapshots. It would basically comb though the snapshot directory, looking for snapshots. If it finds one it hasn't seen, it starts doing the current log splitting on that snapshot (which looks basically like the root directory of hbase - less the ROOT and META tables - so it should be almost, if not entirely, drop-in useable). When the logs are split, we would have to do a little extra checking to make sure that we don't restore a snapshot mid-split, or that if we do that it handles it properly.

          Show
          Jesse Yates added a comment - Another semi-unrelated note... currently we keep full logs files, and the restore needs to split them (see the restore code SnapshotLogSplitter, https://github.com/matteobertozzi/hbase/blob/snapshot-dev/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/restore/RestoreSnapshotHelper.java#L398 ) Can we move this logic at the end of the take snapshot operation and split the logs in .snapshot/region/recover.edits? If we move it into the snapshot operation, then that will slow down the overall operation and make it more difficult to reason about how long a snapshot 'should' take. In particular, this becomes difficult because we want to give the client firm time bounds, but the log splitting is not time bounded (AFAIK). An alternative would be to have a background snapshot-log-splitter task that just goes through and splits logs for snapshots. It would basically comb though the snapshot directory, looking for snapshots. If it finds one it hasn't seen, it starts doing the current log splitting on that snapshot (which looks basically like the root directory of hbase - less the ROOT and META tables - so it should be almost, if not entirely, drop-in useable). When the logs are split, we would have to do a little extra checking to make sure that we don't restore a snapshot mid-split, or that if we do that it handles it properly.
          Hide
          Matteo Bertozzi added a comment -

          Current offline snapshot status:
          the code was up for review for a while now, and everything is at least +1
          we're missing a couple of reviews to merge it in the snapshot branch.

          Jira Description Status Review Link
          HBASE-5547 HFile Archiver trunk  
          HBASE-6610 HFileLink hardlink alternative trunk  
          HBASE-6571 Error handling framework snapshot branch review board
          HBASE-6765 Take a Snapshot Interface snapshot branch review board
          HBASE-6230 Snapshot Reference Utils snapshot branch review board
          HBASE-6353 Snapshot Shell snapshot-branch review board
          HBASE-6863 Offline Snapshot review +2 review board
          HBASE-6865 Snapshot cleaner review +2 review board
          HBASE-6777 Restore Interface review +1 review board
          HBASE-6230 Restore Snapshot review +1 review board
          HBASE-6802 Export Snapshot review +1 review board

          The reference snapshot branch is: https://github.com/jyates/hbase/tree/snapshots
          The "complete" dev branch with all commit above is: https://github.com/matteobertozzi/hbase/commits/offline-snapshot-review-v3

          Show
          Matteo Bertozzi added a comment - Current offline snapshot status: the code was up for review for a while now, and everything is at least +1 we're missing a couple of reviews to merge it in the snapshot branch. Jira Description Status Review Link HBASE-5547 HFile Archiver trunk   HBASE-6610 HFileLink hardlink alternative trunk   HBASE-6571 Error handling framework snapshot branch review board HBASE-6765 Take a Snapshot Interface snapshot branch review board HBASE-6230 Snapshot Reference Utils snapshot branch review board HBASE-6353 Snapshot Shell snapshot-branch review board HBASE-6863 Offline Snapshot review +2 review board HBASE-6865 Snapshot cleaner review +2 review board HBASE-6777 Restore Interface review +1 review board HBASE-6230 Restore Snapshot review +1 review board HBASE-6802 Export Snapshot review +1 review board The reference snapshot branch is: https://github.com/jyates/hbase/tree/snapshots The "complete" dev branch with all commit above is: https://github.com/matteobertozzi/hbase/commits/offline-snapshot-review-v3
          Hide
          Jonathan Hsieh added a comment -

          I think we need to get all those committed to the branch, and then put up a doc about the features, what it provides, how it works, and why we chose particular semantics. We also need to document its current caveats.

          We'll do some testing and probably need to do a little rebasing before we can consider a trunk merge.

          We'll put a flag up here when we are ready but as a heads up, we'd like 1-2 more committers to review. (Currently it is Me, Jesse, and Ted in some places).

          Show
          Jonathan Hsieh added a comment - I think we need to get all those committed to the branch, and then put up a doc about the features, what it provides, how it works, and why we chose particular semantics. We also need to document its current caveats. We'll do some testing and probably need to do a little rebasing before we can consider a trunk merge. We'll put a flag up here when we are ready but as a heads up, we'd like 1-2 more committers to review. (Currently it is Me, Jesse, and Ted in some places).
          Hide
          Jesse Yates added a comment -

          +1 on Jon's comments. I think most of this is almost there

          Show
          Jesse Yates added a comment - +1 on Jon's comments. I think most of this is almost there
          Hide
          Jonathan Hsieh added a comment -

          I've reached a pretty decent point on this branch now refactoring and simplifying the online snapshot code. Most of the work is in the plumbing, and most of the complexity remains in the plumbing. Here is a link

          https://github.com/jmhsieh/hbase/commits/snapshot-dev?

          It isn't worth looking at the individual patches – if interested take a look at the code generally one package at a time. I'd suggest starting from errorhandling, then procedures since this work is fairly isolated and stable now. The will likely affect existing code, and online may change significantly during the merge process.

          Error handling: o.a.h.h.errorhandling.*

          • Added concept of ExternalException (an exception from an separate thread or process).
          • Removing generics by funneling everything through an ExternalException
          • Simplified Exception Propagation by only having a Dispatcher, Listener, and Checker. (No Visitors, Orchestrators, some Factories)
          • Made Exception Serialization static so that instances don't need to be passed around.
          • Added more meaningful usage and motivation documentation.

          Procedure framework: o.a.h.h.procedure.*

          • Separated Coordinator side from Member side
          • Reduced the number of constructors (and fewer objects threaded throughout).
          • Added concept of Procedure and Subprocedure – these maintain state on each host. (this replaces just using strings everywhere).
          • Folded several threads that used latch ping-pong into single threads.
          • Renamed methods from 2pc nomenclature to barrier nomenclature.
          • Added more meaningful usage and motivation documentation.

          Online Snapshots: o.a.h.h.snapshot.*

          • Converted per regionserver only Procedures to simpler Callable/Future fork join implementation.
          • Removed different *ErrorHandlers and moved into Subprocedures. (this may be further eliminated)
          • Each Procedure contains an ExternalExceptionDispatcher
          • ExternalExceptions go to the SnapshotManager to abort the Procedure.

          I'm in process of merging code into the offline snapshot branch. It isn't clean but I'll be working on that for the next few days. (Clashes with updates in offline snapshot). Once I get the snapshot branch compiling again, I'll start posting the External Exception and Procedure stuff as a series of patches.

          My suggestion for the overall effort is to get the main offline snapshot branch code committed to the branch and then start looking into merging with trunk and 0.94. The online work I feel should remain a branch until its pieces are fleshed out.

          Show
          Jonathan Hsieh added a comment - I've reached a pretty decent point on this branch now refactoring and simplifying the online snapshot code. Most of the work is in the plumbing, and most of the complexity remains in the plumbing. Here is a link https://github.com/jmhsieh/hbase/commits/snapshot-dev? It isn't worth looking at the individual patches – if interested take a look at the code generally one package at a time. I'd suggest starting from errorhandling, then procedures since this work is fairly isolated and stable now. The will likely affect existing code, and online may change significantly during the merge process. Error handling: o.a.h.h.errorhandling.* Added concept of ExternalException (an exception from an separate thread or process). Removing generics by funneling everything through an ExternalException Simplified Exception Propagation by only having a Dispatcher, Listener, and Checker. (No Visitors, Orchestrators, some Factories) Made Exception Serialization static so that instances don't need to be passed around. Added more meaningful usage and motivation documentation. Procedure framework: o.a.h.h.procedure.* Separated Coordinator side from Member side Reduced the number of constructors (and fewer objects threaded throughout). Added concept of Procedure and Subprocedure – these maintain state on each host. (this replaces just using strings everywhere). Folded several threads that used latch ping-pong into single threads. Renamed methods from 2pc nomenclature to barrier nomenclature. Added more meaningful usage and motivation documentation. Online Snapshots: o.a.h.h.snapshot.* Converted per regionserver only Procedures to simpler Callable/Future fork join implementation. Removed different *ErrorHandlers and moved into Subprocedures. (this may be further eliminated) Each Procedure contains an ExternalExceptionDispatcher ExternalExceptions go to the SnapshotManager to abort the Procedure. I'm in process of merging code into the offline snapshot branch. It isn't clean but I'll be working on that for the next few days. (Clashes with updates in offline snapshot). Once I get the snapshot branch compiling again, I'll start posting the External Exception and Procedure stuff as a series of patches. My suggestion for the overall effort is to get the main offline snapshot branch code committed to the branch and then start looking into merging with trunk and 0.94. The online work I feel should remain a branch until its pieces are fleshed out.
          Hide
          Jonathan Hsieh added a comment -
          Show
          Jonathan Hsieh added a comment - Branch to a squashed version. https://github.com/jmhsieh/hbase/tree/snapshot-dev-squash
          Hide
          Jesse Yates added a comment -

          I just +1'ed Matteo's restore changes, so I hope to roll that into the snapshot branch (https://github.com/jyates/hbase/tree/snapshots) in the next day or two, which would mean we have everything for offline snapshots.

          The offline snapshots themselves are a self contained bit of code and sizable enough to make it worthwhile to roll it into trunk on its own.

          Let's keep the online keep on a branch until its wrapped up - I'll start taking a look at the online when I have a chance, and look forward to code posted on RB. The progress sounds real sweet Jon!

          TL;DR lets do what Jon suggests above, especially as offline snapshots are basically done and ready to merge into trunk

          Show
          Jesse Yates added a comment - I just +1'ed Matteo's restore changes, so I hope to roll that into the snapshot branch ( https://github.com/jyates/hbase/tree/snapshots ) in the next day or two, which would mean we have everything for offline snapshots. The offline snapshots themselves are a self contained bit of code and sizable enough to make it worthwhile to roll it into trunk on its own. Let's keep the online keep on a branch until its wrapped up - I'll start taking a look at the online when I have a chance, and look forward to code posted on RB. The progress sounds real sweet Jon! TL;DR lets do what Jon suggests above, especially as offline snapshots are basically done and ready to merge into trunk
          Hide
          Jonathan Hsieh added a comment -

          Just force pushed the snapshot-dev-squash branch – it didn't delete files that were supposed to be gone.

          Show
          Jonathan Hsieh added a comment - Just force pushed the snapshot-dev-squash branch – it didn't delete files that were supposed to be gone.
          Hide
          Jonathan Hsieh added a comment -

          Here's a repo with the online branch merged to the snapshot branch with snapshot unit tests passing now.

          https://github.com/jmhsieh/hbase/tree/snapshots-online-merge
          hash d1299347c0c1afcc0264b14ee12beee170efc4c2

          mvn clean
          mvn test -Dtest=errorhandling/* -PlocalTests
          mvn test -Dtest=Test*Procedure* -PlocalTests
          mvn test -Dtest=Test*Snapshot*,snapshot/*,TestFSUtils -PlocalTests

          It needs some cleanup (cleanup duplicate/commented code from merge) but patches of pieces will be coming out later today / tommorrow.

          Likely pieces: ( might break a few more of these down if they are excessive)

          • External Exceptions + snapshot manager refactor
          • Barrier Procedure
          • online Timestamp snapshots tasks
          • online timestampe snapshots + snapshot manager
          Show
          Jonathan Hsieh added a comment - Here's a repo with the online branch merged to the snapshot branch with snapshot unit tests passing now. https://github.com/jmhsieh/hbase/tree/snapshots-online-merge hash d1299347c0c1afcc0264b14ee12beee170efc4c2 mvn clean mvn test -Dtest=errorhandling/* -PlocalTests mvn test -Dtest=Test*Procedure* -PlocalTests mvn test -Dtest=Test*Snapshot*,snapshot/*,TestFSUtils -PlocalTests It needs some cleanup (cleanup duplicate/commented code from merge) but patches of pieces will be coming out later today / tommorrow. Likely pieces: ( might break a few more of these down if they are excessive) External Exceptions + snapshot manager refactor Barrier Procedure online Timestamp snapshots tasks online timestampe snapshots + snapshot manager
          Hide
          Jonathan Hsieh added a comment -

          Jesse Yates Matteo Bertozzi What do you guys about closing this issue when offline makes it into trunk and having a separate umbrella issue for online-snapshots and variants? If we'd do this we'd move some of the remaining subtasks to the new issue.

          I'm also considering closing some of the existing subtasks and creating new issues with the simplified versions – while they have similar purposes the design and implementation details are somewhat different. comments concerns?

          Show
          Jonathan Hsieh added a comment - Jesse Yates Matteo Bertozzi What do you guys about closing this issue when offline makes it into trunk and having a separate umbrella issue for online-snapshots and variants? If we'd do this we'd move some of the remaining subtasks to the new issue. I'm also considering closing some of the existing subtasks and creating new issues with the simplified versions – while they have similar purposes the design and implementation details are somewhat different. comments concerns?
          Hide
          Ted Yu added a comment -

          In that case the title of this JIRA should signify offline snapshots, right ?

          Show
          Ted Yu added a comment - In that case the title of this JIRA should signify offline snapshots, right ?
          Hide
          Jonathan Hsieh added a comment -

          If I get some +1's or no comments after Monday, I'll update it.

          Show
          Jonathan Hsieh added a comment - If I get some +1's or no comments after Monday, I'll update it.
          Hide
          Matteo Bertozzi added a comment -

          +1 on separate offline and online, but maybe we can keep a root jira to keep track of all the dependencies, and a general design doc

           + Snapshot in HBase
           |-- HFile Archiver
           |-- Offline Snapshot
           |----- Offline Snapshot
           |----- Cleaner
           |----- Restore/Clone
           |----- Shell
           |----- ...
           |-- Online Snapshot
           |----- Procedure
           |----- Exception Framework
           |----- Timestamp Snapshot
           |----- ...
          
          Show
          Matteo Bertozzi added a comment - +1 on separate offline and online, but maybe we can keep a root jira to keep track of all the dependencies, and a general design doc + Snapshot in HBase |-- HFile Archiver |-- Offline Snapshot |----- Offline Snapshot |----- Cleaner |----- Restore/Clone |----- Shell |----- ... |-- Online Snapshot |----- Procedure |----- Exception Framework |----- Timestamp Snapshot |----- ...
          Hide
          Jesse Yates added a comment -

          I agree with Matteo wrt Jon's comments. I expect the 'online scaffolding' will have to dramatically change too, though it depends on what Jon comes up with.

          Jonathan Hsieh feel free to close out the old jiras for things that you are replacing (e.g. the Three-Phase Commit Framework), if you feel up to it.

          Show
          Jesse Yates added a comment - I agree with Matteo wrt Jon's comments. I expect the 'online scaffolding' will have to dramatically change too, though it depends on what Jon comes up with. Jonathan Hsieh feel free to close out the old jiras for things that you are replacing (e.g. the Three-Phase Commit Framework), if you feel up to it.
          Hide
          Jesse Yates added a comment -

          Attaching doc that covers a range of information on how offline snapshots and recovery work. It starts out talking a bit about the high-level of each feature, and then does a walk-through of offline snapshots, restore, clone and export.

          This doc describes out general FS layouts, ordering of events, process ownership, and pending concerns.

          Show
          Jesse Yates added a comment - Attaching doc that covers a range of information on how offline snapshots and recovery work. It starts out talking a bit about the high-level of each feature, and then does a walk-through of offline snapshots, restore, clone and export. This doc describes out general FS layouts, ordering of events, process ownership, and pending concerns.
          Hide
          Jonathan Hsieh added a comment -

          Committed as part of the HBASE-7290/HBASE-6055 snapshot branch merge.

          Show
          Jonathan Hsieh added a comment - Committed as part of the HBASE-7290 / HBASE-6055 snapshot branch merge.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3888 (See https://builds.apache.org/job/HBase-TRUNK/3888/)
          HBASE-7290 / HBASE-6055 Online and Offline table snapshots. Merged to trunk. (Jesse Yates, Matteo Bertozzi, Jonathan Hsieh, Ted Yu) (Revision 1448506)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ErrorHandlingProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java
          • /hbase/trunk/hbase-protocol/src/main/protobuf/ErrorHandling.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/MasterAdmin.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/trunk/hbase-server/src/main/avro
          • /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionDispatcher.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionListener.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionSnare.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutExceptionInjector.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/CloneSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/RestoreSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/migration
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Procedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinator.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinatorRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMember.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMemberRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Subprocedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/SubprocedureFactory.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureCoordinatorRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureMemberRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CopyRecoveredEditsTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CorruptedSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/HBaseSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceRegionHFilesTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceServerWALsTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotCreationException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDoesNotExistException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotExistsException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TakeSnapshotUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/UnknownSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/snapshot.jsp
          • /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb
          • /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/clone_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/delete_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list_snapshots.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/rename_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/restore_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/snapshot.rb
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotMetadata.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/metrics
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/migration
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/metrics
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3888 (See https://builds.apache.org/job/HBase-TRUNK/3888/ ) HBASE-7290 / HBASE-6055 Online and Offline table snapshots. Merged to trunk. (Jesse Yates, Matteo Bertozzi, Jonathan Hsieh, Ted Yu) (Revision 1448506) Result = FAILURE jmhsieh : Files : /hbase/trunk /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ErrorHandlingProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java /hbase/trunk/hbase-protocol/src/main/protobuf/ErrorHandling.proto /hbase/trunk/hbase-protocol/src/main/protobuf/MasterAdmin.proto /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/main/avro /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionDispatcher.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionListener.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionSnare.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutExceptionInjector.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/CloneSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/RestoreSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/migration /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Procedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinator.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinatorRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMember.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMemberRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Subprocedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/SubprocedureFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureCoordinatorRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureMemberRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CopyRecoveredEditsTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CorruptedSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/HBaseSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceRegionHFilesTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceServerWALsTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotCreationException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDoesNotExistException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotExistsException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TakeSnapshotUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/UnknownSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/snapshot.jsp /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb /hbase/trunk/hbase-server/src/main/ruby/shell.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/clone_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/delete_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list_snapshots.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/rename_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/restore_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/snapshot.rb /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotMetadata.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/metrics /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/migration /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/metrics /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #414 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/414/)
          HBASE-7290 / HBASE-6055 Online and Offline table snapshots. Merged to trunk. (Jesse Yates, Matteo Bertozzi, Jonathan Hsieh, Ted Yu) (Revision 1448506)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk
          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ErrorHandlingProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
          • /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java
          • /hbase/trunk/hbase-protocol/src/main/protobuf/ErrorHandling.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/MasterAdmin.proto
          • /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto
          • /hbase/trunk/hbase-server/src/main/avro
          • /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionDispatcher.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionListener.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionSnare.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutExceptionInjector.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/CloneSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/RestoreSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/migration
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Procedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinator.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinatorRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMember.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMemberRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Subprocedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/SubprocedureFactory.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureCoordinatorRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureMemberRpcs.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/metrics
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CopyRecoveredEditsTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CorruptedSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/HBaseSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceRegionHFilesTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceServerWALsTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotCreationException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDoesNotExistException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotExistsException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TakeSnapshotUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/UnknownSnapshotException.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/snapshot.jsp
          • /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb
          • /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/clone_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/delete_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list_snapshots.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/rename_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/restore_snapshot.rb
          • /hbase/trunk/hbase-server/src/main/ruby/shell/commands/snapshot.rb
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotMetadata.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/metrics
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/migration
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/metrics
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #414 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/414/ ) HBASE-7290 / HBASE-6055 Online and Offline table snapshots. Merged to trunk. (Jesse Yates, Matteo Bertozzi, Jonathan Hsieh, Ted Yu) (Revision 1448506) Result = FAILURE jmhsieh : Files : /hbase/trunk /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ErrorHandlingProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java /hbase/trunk/hbase-protocol/src/main/protobuf/ErrorHandling.proto /hbase/trunk/hbase-protocol/src/main/protobuf/MasterAdmin.proto /hbase/trunk/hbase-protocol/src/main/protobuf/hbase.proto /hbase/trunk/hbase-server/src/main/avro /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionDispatcher.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionListener.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/ForeignExceptionSnare.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/errorhandling/TimeoutExceptionInjector.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DisableTableHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/CloneSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/RestoreSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/migration /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Procedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinator.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureCoordinatorRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMember.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMemberRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Subprocedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/SubprocedureFactory.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureCoordinatorRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureMemberRpcs.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ZKProcedureUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/FlushSnapshotSubprocedure.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/snapshot/RegionServerSnapshotManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/metrics /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CopyRecoveredEditsTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CorruptedSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/HBaseSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceRegionHFilesTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceServerWALsTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotCreationException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDoesNotExistException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotExistsException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TablePartiallyOpenException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TakeSnapshotUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/UnknownSnapshotException.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/snapshot.jsp /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb /hbase/trunk/hbase-server/src/main/ruby/shell.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/clone_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/delete_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/list_snapshots.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/rename_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/restore_snapshot.rb /hbase/trunk/hbase-server/src/main/ruby/shell/commands/snapshot.rb /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromAdmin.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotMetadata.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionDispatcher.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestForeignExceptionSerialization.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/errorhandling/TestTimeoutExceptionInjector.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/metrics /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotLogCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/migration /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedure.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureCoordinator.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureMember.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedureControllers.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/metrics /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jesse Yates
            • Votes:
              2 Vote for this issue
              Watchers:
              42 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development