HBase
  1. HBase
  2. HBASE-5547

Don't delete HFiles when in "backup mode"

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      All HFiles are now automatically archived to the configured "hbase.table.archive.directory". HFiles are cleaned similarly to HLogs - a cleaner delegate chain that is instantiated as per the configured classes. Similar to hlog cleaners, a TimeToLiveHFileCleaner is used by default, and specified under the "hbase.master.hfilecleaner.plugins" configuration key.

      In unifying the two cleaner interfaces, the new configuration parameter to use the TimeToLiveHLogCleaner is:
        <property>
          <name>hbase.master.logcleaner.plugins</name>
          <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner</value>
       </property>

      It is still enabled by default, so nothing needs to be changed if you are not modifying the logcleaners.
      Show
      All HFiles are now automatically archived to the configured "hbase.table.archive.directory". HFiles are cleaned similarly to HLogs - a cleaner delegate chain that is instantiated as per the configured classes. Similar to hlog cleaners, a TimeToLiveHFileCleaner is used by default, and specified under the "hbase.master.hfilecleaner.plugins" configuration key. In unifying the two cleaner interfaces, the new configuration parameter to use the TimeToLiveHLogCleaner is:   <property>     <name>hbase.master.logcleaner.plugins</name>     <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner</value>  </property> It is still enabled by default, so nothing needs to be changed if you are not modifying the logcleaners.

      Description

      This came up in a discussion I had with Stack.
      It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either:
      1. rename HFiles to be delete to <file>.bck
      2. rename the HFiles into a special directory
      3. rename them to a general trash directory (which would not need to be tied to backup mode).

      That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those).

      #1 makes cleanup a bit harder.

      1. hbase-5547-0.94-backport-v0.patch
        147 kB
        Jesse Yates
      2. 5547-addendum-v4.txt
        0.7 kB
        Ted Yu
      3. 5547.addendum-v3
        7 kB
        Ted Yu
      4. java_HBASE-5547.addendum-v2
        7 kB
        Jesse Yates
      5. java_HBASE-5547.addendum-v1
        13 kB
        Jesse Yates
      6. java_HBASE-5547.addendum
        7 kB
        Jesse Yates
      7. 5547-v16.txt
        196 kB
        Ted Yu
      8. java_HBASE-5547_v15.patch
        196 kB
        Ted Yu
      9. java_HBASE-5547_v14.patch
        194 kB
        Jesse Yates
      10. java_HBASE-5547_v13.patch
        190 kB
        Jesse Yates
      11. 5547-v12.txt
        184 kB
        Ted Yu
      12. hbase-5547-v9.patch
        185 kB
        Jesse Yates
      13. hbase-5447-v8.patch
        168 kB
        Jesse Yates
      14. hbase-5447-v8.patch
        168 kB
        Ted Yu
      15. java_HBASE-5547_v7.patch
        162 kB
        Jesse Yates
      16. java_HBASE-5547_v6.patch
        195 kB
        Jesse Yates
      17. java_HBASE-5547_v5.patch
        195 kB
        Jesse Yates
      18. java_HBASE-5547_v4.patch
        195 kB
        Jesse Yates

        Issue Links

          Activity

          Hide
          Jesse Yates added a comment -

          Linking to HBASE-50 as we are also going to have to keep around hfiles indefinitely for that functionality.

          Show
          Jesse Yates added a comment - Linking to HBASE-50 as we are also going to have to keep around hfiles indefinitely for that functionality.
          Hide
          Jesse Yates added a comment -

          Thoughts on how long we should keep around files? Indefinitely? The latter seems a bit excessive, especially if a 'backup mode' ensures you run every X minutes (and exports to another cluster, moves the files to another backup directory). 'Cleanup' in implies you want to remove the file when no one care about the hfiles anymore - thinking maybe a periodic chore on the rs?

          With snapshots, I was expecting to add an file reference feature - essentially doing impl hardlinks for files we care about keeping around. Was thinking we could add a CP hook and impl that would let you add a checks (config based?) for if you want to keep a reference around for the file being cleaned up. In the backup situation, you would have a timer or (maybe check for a backup completed file/meta row) and see if you had elapsed that time or not; if not, you would add a reference, if so, do nothing and let the file get cleaned up.

          Show
          Jesse Yates added a comment - Thoughts on how long we should keep around files? Indefinitely? The latter seems a bit excessive, especially if a 'backup mode' ensures you run every X minutes (and exports to another cluster, moves the files to another backup directory). 'Cleanup' in implies you want to remove the file when no one care about the hfiles anymore - thinking maybe a periodic chore on the rs? With snapshots, I was expecting to add an file reference feature - essentially doing impl hardlinks for files we care about keeping around. Was thinking we could add a CP hook and impl that would let you add a checks (config based?) for if you want to keep a reference around for the file being cleaned up. In the backup situation, you would have a timer or (maybe check for a backup completed file/meta row) and see if you had elapsed that time or not; if not, you would add a reference, if so, do nothing and let the file get cleaned up.
          Hide
          Lars Hofhansl added a comment -

          Was thinking that these files are up grabs unless HBase is in "backup mode", maybe by the existence of a specific zNode, although other models are possible as well.
          The details are a bit tricky, of course. Do we need a full cleanup between backup runs so that we do not confuse the backup files? If not, do we tag with a backup number or a with timestamp (like the HLogs)?

          If we do HBASE-50 we won't need this one methinks. This might get us to workable solution more quickly.

          Show
          Lars Hofhansl added a comment - Was thinking that these files are up grabs unless HBase is in "backup mode", maybe by the existence of a specific zNode, although other models are possible as well. The details are a bit tricky, of course. Do we need a full cleanup between backup runs so that we do not confuse the backup files? If not, do we tag with a backup number or a with timestamp (like the HLogs)? If we do HBASE-50 we won't need this one methinks. This might get us to workable solution more quickly.
          Hide
          stack added a comment -

          Thoughts on how long we should keep around files? Indefinitely?

          Yes. I'd think some other process other than hbase would be responsible for their cleanup.

          Show
          stack added a comment - Thoughts on how long we should keep around files? Indefinitely? Yes. I'd think some other process other than hbase would be responsible for their cleanup.
          Hide
          Jesse Yates added a comment -

          HBASE-50 would build on this functionality (just getting to where I need it anyways), so thought I could knock this out the 'right' way and then later wrap it into hbase-50. The reference stuff is a bit involved and I would be fine if we just moved the files out to another directory. Backup then can just check file times and remove the files that were backed up since the time or copy the ones that are currently being used.

          Timestamp can just looked up per backed-up file or possibly grouped by directory (though the latter seems likely to fragmentation.

          HBase is in "backup mode", maybe by the existence of a specific zNode, although other models are possible as well.

          Like this idea - only backing up while the switch is on, but otherwise should help keep cruft down (not be part of normal operation).

          Show
          Jesse Yates added a comment - HBASE-50 would build on this functionality (just getting to where I need it anyways), so thought I could knock this out the 'right' way and then later wrap it into hbase-50. The reference stuff is a bit involved and I would be fine if we just moved the files out to another directory. Backup then can just check file times and remove the files that were backed up since the time or copy the ones that are currently being used. Timestamp can just looked up per backed-up file or possibly grouped by directory (though the latter seems likely to fragmentation. HBase is in "backup mode", maybe by the existence of a specific zNode, although other models are possible as well. Like this idea - only backing up while the switch is on, but otherwise should help keep cruft down (not be part of normal operation).
          Hide
          Jesse Yates added a comment -

          Implementation consideration: should we be backing up all deletes or just those that are system enabled. For instance, we could delete a table via the api/shell. However, that is a pointed act and probably doesn't mean we want to back that data up. Proposing we keep it simple FTM and just backup system removed HFiles. Otherwise, we keep around tons of copies of files.

          Alternative I wouldn't mind (but increases the size of the patch and feels like scope creep) would be to backup even for table level deletes and on things like deleting a table, you need a bool to specify "but not the backup".

          thoughts?

          Show
          Jesse Yates added a comment - Implementation consideration: should we be backing up all deletes or just those that are system enabled. For instance, we could delete a table via the api/shell. However, that is a pointed act and probably doesn't mean we want to back that data up. Proposing we keep it simple FTM and just backup system removed HFiles. Otherwise, we keep around tons of copies of files. Alternative I wouldn't mind (but increases the size of the patch and feels like scope creep) would be to backup even for table level deletes and on things like deleting a table, you need a bool to specify "but not the backup". thoughts?
          Hide
          Lars Hofhansl added a comment -

          I'd just backup system removed HFiles if you can always tell the difference.

          Show
          Lars Hofhansl added a comment - I'd just backup system removed HFiles if you can always tell the difference.
          Hide
          stack added a comment -

          What Lars said.

          Show
          stack added a comment - What Lars said.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/
          -----------------------------------------------------------

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary
          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:
          Updated the HMasterInterface to pass the calls onto the zookeeper.
          Added a zk listener to handle updates from the master to the RS to backup.
          Added a utility for removing files and finding archive directories
          Added tests for the regionserver and catalogjanitor approaches.
          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.
          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs


          src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace
          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655
          src/main/resources/hbase-default.xml 341431a
          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7

          Diff: https://reviews.apache.org/r/4633/diff

          Testing
          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/resources/hbase-default.xml 341431a src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6672
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/HConstants.java
          <https://reviews.apache.org/r/4633/#comment14456>

          There is no mentioning of HFile in the name of znode.
          What if another archive scheme comes up in the future ?

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          <https://reviews.apache.org/r/4633/#comment14433>

          What is the counterpart for disableHFileBackup() where backup for all tables is enabled ?

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14435>

          License.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14436>

          'or not' is not needed.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14437>

          Remove empty line.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment14438>

          white space.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment14441>

          Should we log if rss == null ?
          Since in that case we cannot perform desired archiving.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment14439>

          Should they be named hFileArchive

          {Manager,Tracker}

          ?

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          <https://reviews.apache.org/r/4633/#comment14440>

          The leading 'get ' is not necessary.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14442>

          License.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14443>

          Please check the return value from mkdirs().

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14444>

          Please check the return value.
          Should resolveAndCopyFile() be enclosed in a try block ?

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14445>

          The code at line 92 doesn't 'keep' HFiles.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14446>

          This doesn't match the fs.rename() call.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14447>

          What about non-directory files ?

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14448>

          Should include file in log message.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14449>

          Move this to line 197.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14450>

          Should this be contingent on success being true ?

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14451>

          Move else if to line 219 and enclose LOG.debug in curly braces.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14434>

          Add license, please.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14452>

          Should this method return a boolean which indicates whether the tracker is enabled ?

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14453>

          variable table should be included in the log.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14454>

          Remove first 'the'.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14455>

          Why do we return when archiveHFileZNode exists ?

          • Ted

          On 2012-04-04 01:23:29, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-04 01:23:29)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655

          src/main/resources/hbase-default.xml 341431a

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6672 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/HConstants.java < https://reviews.apache.org/r/4633/#comment14456 > There is no mentioning of HFile in the name of znode. What if another archive scheme comes up in the future ? src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java < https://reviews.apache.org/r/4633/#comment14433 > What is the counterpart for disableHFileBackup() where backup for all tables is enabled ? src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14435 > License. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14436 > 'or not' is not needed. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14437 > Remove empty line. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment14438 > white space. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment14441 > Should we log if rss == null ? Since in that case we cannot perform desired archiving. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment14439 > Should they be named hFileArchive {Manager,Tracker} ? src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java < https://reviews.apache.org/r/4633/#comment14440 > The leading 'get ' is not necessary. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14442 > License. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14443 > Please check the return value from mkdirs(). src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14444 > Please check the return value. Should resolveAndCopyFile() be enclosed in a try block ? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14445 > The code at line 92 doesn't 'keep' HFiles. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14446 > This doesn't match the fs.rename() call. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14447 > What about non-directory files ? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14448 > Should include file in log message. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14449 > Move this to line 197. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14450 > Should this be contingent on success being true ? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14451 > Move else if to line 219 and enclose LOG.debug in curly braces. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14434 > Add license, please. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14452 > Should this method return a boolean which indicates whether the tracker is enabled ? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14453 > variable table should be included in the log. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14454 > Remove first 'the'. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14455 > Why do we return when archiveHFileZNode exists ? Ted On 2012-04-04 01:23:29, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-04 01:23:29) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/resources/hbase-default.xml 341431a src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          Should we consider the duration for HFile retention ?
          Unlimited backup of HFiles for big table(s) would some day fill up hdfs.

          Show
          Ted Yu added a comment - Should we consider the duration for HFile retention ? Unlimited backup of HFiles for big table(s) would some day fill up hdfs.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6714
          -----------------------------------------------------------

          Thanks for the review Ted. Any thoughts on combining the Tracker and Manager (see inline comment below on HFileArchiveManager).

          src/main/java/org/apache/hadoop/hbase/HConstants.java
          <https://reviews.apache.org/r/4633/#comment14522>

          fair enough. done.

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          <https://reviews.apache.org/r/4633/#comment14517>

          Disable right now just wipes out all the tables currently in backup mode (not saving any info), but I guess it could be reasonable to want to start backing up all the tables - though in real usage you probably only want to do one table at a time to minimize impact on a cluster.

          I'd rather not give people that power directly, but make them jump through a couple hoops (listing out all the tables and then iteratively starting backups on those tables) due: 1) how much data that can be, and 2) in a real 'backup' context this means they are cloning all the tables (doing a full table scan on all tables) which would be highly disruptive.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14518>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14519>

          done

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14520>

          I've been debating whether or not to roll this implementation into the HFileArchiveTracker and just move this into an interface. It doesn't actually do very much - essentially a prettier front to a map and could easily be moved into the tracker.

          Thoughts?

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14521>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment14526>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment14523>

          That case should only come up in testing, so logging seems excessive.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment14524>

          used store because in code that's what we are really dealing with, but everyone just refers to them as hfiles. going with your way for consistency.

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          <https://reviews.apache.org/r/4633/#comment14525>

          done.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14527>

          done.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14528>

          yup, nice catch (no pun intended).

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14529>

          true. earlier iteration comment that got confused with manager.keepFiles

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14530>

          how so? renames an the src path to the dest path. Seems to work in tests.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14531>

          then its something we don't care about - only care about the hfiles, which are stored in one directory level below the region.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14532>

          done.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14534>

          Definitely need to add another unit test covering the 'resolve' part of this method - lot of gymnastics/corner cases going on.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14536>

          reworked a little bit in the new version.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14533>

          going to bail out earlier if we can't delete the old file.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14535>

          done.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14537>

          done.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14538>

          don't think so - enabled only means that it found the archive znode, in which case it will attempt to update the manager. The enabled construct is really an internal util, as opposed to something that anyone else should really care about (at least that was the intention.)

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14539>

          oops, shouldn't have. Adding unit test to catch that.

          • Jesse

          On 2012-04-04 01:23:29, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-04 01:23:29)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655

          src/main/resources/hbase-default.xml 341431a

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6714 ----------------------------------------------------------- Thanks for the review Ted. Any thoughts on combining the Tracker and Manager (see inline comment below on HFileArchiveManager). src/main/java/org/apache/hadoop/hbase/HConstants.java < https://reviews.apache.org/r/4633/#comment14522 > fair enough. done. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java < https://reviews.apache.org/r/4633/#comment14517 > Disable right now just wipes out all the tables currently in backup mode (not saving any info), but I guess it could be reasonable to want to start backing up all the tables - though in real usage you probably only want to do one table at a time to minimize impact on a cluster. I'd rather not give people that power directly, but make them jump through a couple hoops (listing out all the tables and then iteratively starting backups on those tables) due: 1) how much data that can be, and 2) in a real 'backup' context this means they are cloning all the tables (doing a full table scan on all tables) which would be highly disruptive. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14518 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14519 > done src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14520 > I've been debating whether or not to roll this implementation into the HFileArchiveTracker and just move this into an interface. It doesn't actually do very much - essentially a prettier front to a map and could easily be moved into the tracker. Thoughts? src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14521 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment14526 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment14523 > That case should only come up in testing, so logging seems excessive. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment14524 > used store because in code that's what we are really dealing with, but everyone just refers to them as hfiles. going with your way for consistency. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java < https://reviews.apache.org/r/4633/#comment14525 > done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14527 > done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14528 > yup, nice catch (no pun intended). src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14529 > true. earlier iteration comment that got confused with manager.keepFiles src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14530 > how so? renames an the src path to the dest path. Seems to work in tests. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14531 > then its something we don't care about - only care about the hfiles, which are stored in one directory level below the region. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14532 > done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14534 > Definitely need to add another unit test covering the 'resolve' part of this method - lot of gymnastics/corner cases going on. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14536 > reworked a little bit in the new version. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14533 > going to bail out earlier if we can't delete the old file. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14535 > done. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14537 > done. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14538 > don't think so - enabled only means that it found the archive znode, in which case it will attempt to update the manager. The enabled construct is really an internal util, as opposed to something that anyone else should really care about (at least that was the intention.) src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14539 > oops, shouldn't have. Adding unit test to catch that. Jesse On 2012-04-04 01:23:29, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-04 01:23:29) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/resources/hbase-default.xml 341431a src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          I currently don't have good idea w.r.t. combining HFileArchiveManager with the tracker.
          One of them would be exposed to HRegion anyway.

          Show
          Ted Yu added a comment - I currently don't have good idea w.r.t. combining HFileArchiveManager with the tracker. One of them would be exposed to HRegion anyway.
          Hide
          Jesse Yates added a comment -
          {qoute}

          Should we consider the duration for HFile retention ?
          Unlimited backup of HFiles for big table(s) would some day fill up hdfs.

          Discussed above that they should be kept around indefinitely. Thought is that you enable archiving, run your backup process which also cleans out the hfiles when they aren't needed, and then turns off archiving.

          Show
          Jesse Yates added a comment - {qoute} Should we consider the duration for HFile retention ? Unlimited backup of HFiles for big table(s) would some day fill up hdfs. Discussed above that they should be kept around indefinitely. Thought is that you enable archiving, run your backup process which also cleans out the hfiles when they aren't needed, and then turns off archiving.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/
          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11.515830)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Changes
          -------

          A bit rearchitecting in the new patch:
          1) Separated the client and server archiving into distinct classes (and saved changes to HMasterInferface)
          2) A lot of the functions in the ArchiveUtil were moved to their respective classes and move a couple smaller methods in
          3) Added a synchronous enable as the default for the client
          4) Merged the tracker and the manger into the tracker and pulled out the important methods into a monitor interface.
          5) Addressed Ted's comments from the last patch (and hopefully didn't add them back in the change).
          6) Added a bunch of testing

          Summary
          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:
          Updated the HMasterInterface to pass the calls onto the zookeeper.
          Added a zk listener to handle updates from the master to the RS to backup.
          Added a utility for removing files and finding archive directories
          Added tests for the regionserver and catalogjanitor approaches.
          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.
          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017
          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd
          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f
          src/main/resources/hbase-default.xml 44ee689
          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing
          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11.515830) Review request for hbase, Michael Stack and Lars Hofhansl. Changes ------- A bit rearchitecting in the new patch: 1) Separated the client and server archiving into distinct classes (and saved changes to HMasterInferface) 2) A lot of the functions in the ArchiveUtil were moved to their respective classes and move a couple smaller methods in 3) Added a synchronous enable as the default for the client 4) Merged the tracker and the manger into the tracker and pulled out the important methods into a monitor interface. 5) Addressed Ted's comments from the last patch (and hopefully didn't add them back in the change). 6) Added a bunch of testing Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs (updated) src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6777
          -----------------------------------------------------------

          In the javadoc, 'backups' were used. I think backup should be good enough.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14940>

          How about naming this method createHFileArchiveManager or buildHFileArchiveManager ?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14941>

          The directory is specified by
          HConstants.HFILE_ARCHIVE_DIRECTORY.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14943>

          Should read 'all online region servers hosting the'

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14944>

          Typo: received.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14942>

          Please add javadoc for archive.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14945>

          This would create a manager.
          Can we cache the archive manager ?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14952>

          Parameter for allRegionsDoneBackup() is called confirmedDone
          This seems to be inconsistent with the return from regionsArchivingTable()

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14947>

          This call would leave the backup for tableName in an inconsistent state, right ?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14946>

          tableName should be included.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment14948>

          Please mention this call is asynchronous.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14954>

          License.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14949>

          Should read 'archiving for the table'

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment14950>

          How about naming this method regionsBeingArchived or regionsBeingArchivedForTable ?
          I think the first should be enough since table is a parameter.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment14955>

          No year, please.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment14956>

          This doesn't sound consistent with the class being a monitor.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14957>

          No year.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment14959>

          Year is not needed.

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment14960>

          License.

          • Ted

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6777 ----------------------------------------------------------- In the javadoc, 'backups' were used. I think backup should be good enough. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14940 > How about naming this method createHFileArchiveManager or buildHFileArchiveManager ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14941 > The directory is specified by HConstants.HFILE_ARCHIVE_DIRECTORY. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14943 > Should read 'all online region servers hosting the' src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14944 > Typo: received. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14942 > Please add javadoc for archive. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14945 > This would create a manager. Can we cache the archive manager ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14952 > Parameter for allRegionsDoneBackup() is called confirmedDone This seems to be inconsistent with the return from regionsArchivingTable() src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14947 > This call would leave the backup for tableName in an inconsistent state, right ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14946 > tableName should be included. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment14948 > Please mention this call is asynchronous. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14954 > License. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14949 > Should read 'archiving for the table' src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment14950 > How about naming this method regionsBeingArchived or regionsBeingArchivedForTable ? I think the first should be enough since table is a parameter. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment14955 > No year, please. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment14956 > This doesn't sound consistent with the class being a monitor. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14957 > No year. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment14959 > Year is not needed. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment14960 > License. Ted On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6778
          -----------------------------------------------------------

          This is pretty nice!
          Did a first pass. The only part I a bit hard to comprehend is the archiveDir vs. the timestamped archiveDir.

          Somebody with more knowledge about ZK should look through he ZK code. For example do we need to call sync() in some cases to ensure we get the latest ZK view?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment14951>

          Is this a warning?

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment14953>

          How can that be the case? Can we get a new file with the same name as a previously deleted one?
          This might make it hard to piece together the set of files required for backup.

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment14958>

          License missing

          • Lars

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6778 ----------------------------------------------------------- This is pretty nice! Did a first pass. The only part I a bit hard to comprehend is the archiveDir vs. the timestamped archiveDir. Somebody with more knowledge about ZK should look through he ZK code. For example do we need to call sync() in some cases to ensure we get the latest ZK view? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment14951 > Is this a warning? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment14953 > How can that be the case? Can we get a new file with the same name as a previously deleted one? This might make it hard to piece together the set of files required for backup. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment14958 > License missing Lars On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          In latest patch, regions participating in archiving are listed under table znode.
          How are newly split regions added to the tree ?

          Please add a test case for this scenario.

          Show
          Ted Yu added a comment - In latest patch, regions participating in archiving are listed under table znode. How are newly split regions added to the tree ? Please add a test case for this scenario.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6820
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15202>

          done. Originally modeled after the catalog tracker in naming.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15203>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15204>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15205>

          done

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15206>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15207>

          Would rather not, since in that case we would keep around a persistent connection to ZK, which is against what the community has been doing with cutting the client/zk connection (though admittedly that not directly applicable here).

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15208>

          changed to confirmedArchiving.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15184>

          Yeah, it would. But then you should not attempt to expect the archive as valid and fail your backup process. Was going to leave it to the user to determine if they wanted to do the cleanup.

          However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started.

          Unfortunately, this means add an extra method on the master which I was trying to avoid, but I guess its not the end of the world. We also have a two-phase commit issue here as we also need to then write a 'consistent-confirmed' file into the archive when it completes to ensure that failed clients don't break the archiving.

          thoughts?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15209>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15183>

          Method name isn't enough?

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15210>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15211>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15212>

          done - regionsBeingArchived it is.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment15213>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment15185>

          Cruft from the earlier version.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment15186>

          yeah, guess it should be.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment15214>

          done.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment15187>

          This is a super rare case where the name of the regiondir to be backed up is the same as the one in the backup already. Just being extra careful that the random storefile name doesn't match, though it is very unlikely to happen, it still could (which could lead to bad circumstances). Currently, the storefile check doesn't check the archive directory and I think it would be pretty big overhead to expect it to do that as well (lots of added dependencies, calls, etc). We can just do the check and renaming here - its not super complicated and seems to work.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment15215>

          done.

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment15216>

          done.

          • Jesse

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6820 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15202 > done. Originally modeled after the catalog tracker in naming. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15203 > done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15204 > done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15205 > done src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15206 > done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15207 > Would rather not, since in that case we would keep around a persistent connection to ZK, which is against what the community has been doing with cutting the client/zk connection (though admittedly that not directly applicable here). src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15208 > changed to confirmedArchiving. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15184 > Yeah, it would. But then you should not attempt to expect the archive as valid and fail your backup process. Was going to leave it to the user to determine if they wanted to do the cleanup. However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started. Unfortunately, this means add an extra method on the master which I was trying to avoid, but I guess its not the end of the world. We also have a two-phase commit issue here as we also need to then write a 'consistent-confirmed' file into the archive when it completes to ensure that failed clients don't break the archiving. thoughts? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15209 > done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15183 > Method name isn't enough? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15210 > done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15211 > done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15212 > done - regionsBeingArchived it is. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment15213 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment15185 > Cruft from the earlier version. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment15186 > yeah, guess it should be. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment15214 > done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment15187 > This is a super rare case where the name of the regiondir to be backed up is the same as the one in the backup already. Just being extra careful that the random storefile name doesn't match, though it is very unlikely to happen, it still could (which could lead to bad circumstances). Currently, the storefile check doesn't check the archive directory and I think it would be pretty big overhead to expect it to do that as well (lots of added dependencies, calls, etc). We can just do the check and renaming here - its not super complicated and seems to work. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment15215 > done. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment15216 > done. Jesse On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started.

          I think we should do the cleanup if archiving cannot complete.
          If a new method in master interface is needed, we can accommodate that.

          Show
          Ted Yu added a comment - However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started. I think we should do the cleanup if archiving cannot complete. If a new method in master interface is needed, we can accommodate that.
          Hide
          Jesse Yates added a comment -

          @Ted - fair enough. Tried to avoid modifying interfaces, but I guess its not avoidable. Plan is to on failure fire off a call (probably synchronous) to the master to cleanup the FS for that table backup; first writes an 'invalid' marker for that table directory and then recursively deletes the regions and then the table directory in the archive. We should be able to return after writing the invalid marker as it would get cleaned up at some point. Associated unit tests would be: general table archive failure, master died mid-archive failure and recovery cleanup.

          Also thinking about removing the per-table archive in favor of an overall archive directory oriented to tables (and dropping a choice of which directory to archive, in favor of just using the config value). This makes particular sense for snapshots, but is also nice to ensure easier lookups of table values:
          /hbase/.archive/[table]

          rather than the current:
          /hbase/[table]/.archive

          where .archive could actually be any directory the user specified on creation time, which can easily lead to 'loss by forgetfulness'.

          Show
          Jesse Yates added a comment - @Ted - fair enough. Tried to avoid modifying interfaces, but I guess its not avoidable. Plan is to on failure fire off a call (probably synchronous) to the master to cleanup the FS for that table backup; first writes an 'invalid' marker for that table directory and then recursively deletes the regions and then the table directory in the archive. We should be able to return after writing the invalid marker as it would get cleaned up at some point. Associated unit tests would be: general table archive failure, master died mid-archive failure and recovery cleanup. Also thinking about removing the per-table archive in favor of an overall archive directory oriented to tables (and dropping a choice of which directory to archive, in favor of just using the config value). This makes particular sense for snapshots, but is also nice to ensure easier lookups of table values: /hbase/.archive/ [table] rather than the current: /hbase/ [table] /.archive where .archive could actually be any directory the user specified on creation time, which can easily lead to 'loss by forgetfulness'.
          Hide
          Ted Yu added a comment -

          For the recovery cleanup, can we remember the start time of archiving and only delete archive files which are written after this start time ?
          I like the /hbase/.archive/[table] structure.

          Show
          Ted Yu added a comment - For the recovery cleanup, can we remember the start time of archiving and only delete archive files which are written after this start time ? I like the /hbase/.archive/ [table] structure.
          Hide
          Jesse Yates added a comment -

          For the recovery cleanup, can we remember the start time of archiving and only delete archive files which are written after this start time

          totally. I was actually thinking this was the way to go about it last night and completely forgot it this morning. Thanks for looking out

          Show
          Jesse Yates added a comment - For the recovery cleanup, can we remember the start time of archiving and only delete archive files which are written after this start time totally. I was actually thinking this was the way to go about it last night and completely forgot it this morning. Thanks for looking out
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6902
          -----------------------------------------------------------

          Patch looks good. Some concerns in the below.

          src/main/java/org/apache/hadoop/hbase/HConstants.java
          <https://reviews.apache.org/r/4633/#comment15341>

          Do these defines need to be global? Could they be package scoped?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15342>

          What is one of these? Might say so in here.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15343>

          Good doc.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15344>

          Are you missing a version that takes a String?

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15345>

          Does it have to be public? Can it be only of this package?

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15346>

          Is there a close on this class so we can cleanup the zkw?

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15347>

          Should this stuff be generally accessible?

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment15348>

          We are in master package. Why reach over into regionserver package?

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment15349>

          Can hfilemanager be an interface so that when standing up a region to test, we don't have to know much about this type?'

          hmm... I see it is an Interface. Good. Thanks.

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment15350>

          This thing is watching a per table config? Other per table configs are whether a table is enabled or not. Should we coalesce? This is also a configuration maintained via another system, one that persists to zk rather than to an hbase-site.xml or into an HTableDescriptor. Why not go into an htabledescriptor? Nicolas just made it dynamically configurable though the table is rolled

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment15351>

          Does it need to be public? I see master reaches over to get this class so for that its needed but maybe it coudl be packaged elsewhere?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment15352>

          Do you have to stop this on way out?

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment15354>

          Why a ref into regionserver package?

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment15353>

          Whats this class do?

          Should it be abstract since others subclass it?

          Does it need to be public?

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment15355>

          Why these references? Should this tracker be in regionserver package? (Though seems like we are collecting trackers here)

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          <https://reviews.apache.org/r/4633/#comment15356>

          I think its bad that we do this.

          Not your fault.

          • Michael

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6902 ----------------------------------------------------------- Patch looks good. Some concerns in the below. src/main/java/org/apache/hadoop/hbase/HConstants.java < https://reviews.apache.org/r/4633/#comment15341 > Do these defines need to be global? Could they be package scoped? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15342 > What is one of these? Might say so in here. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15343 > Good doc. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15344 > Are you missing a version that takes a String? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15345 > Does it have to be public? Can it be only of this package? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15346 > Is there a close on this class so we can cleanup the zkw? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15347 > Should this stuff be generally accessible? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment15348 > We are in master package. Why reach over into regionserver package? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment15349 > Can hfilemanager be an interface so that when standing up a region to test, we don't have to know much about this type?' hmm... I see it is an Interface. Good. Thanks. src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment15350 > This thing is watching a per table config? Other per table configs are whether a table is enabled or not. Should we coalesce? This is also a configuration maintained via another system, one that persists to zk rather than to an hbase-site.xml or into an HTableDescriptor. Why not go into an htabledescriptor? Nicolas just made it dynamically configurable though the table is rolled src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment15351 > Does it need to be public? I see master reaches over to get this class so for that its needed but maybe it coudl be packaged elsewhere? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment15352 > Do you have to stop this on way out? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment15354 > Why a ref into regionserver package? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment15353 > Whats this class do? Should it be abstract since others subclass it? Does it need to be public? src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment15355 > Why these references? Should this tracker be in regionserver package? (Though seems like we are collecting trackers here) src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java < https://reviews.apache.org/r/4633/#comment15356 > I think its bad that we do this. Not your fault. Michael On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6914
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15366>

          Can the checks in the while loop be reversed, so that we check the counter first and then call the allRegionsDoneBackup(). Will this help in removing the call to disableHFileBackup, or am i missing anything here.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15363>

          yeah, instantiation of this call is done more than once in a single code path (call to HBaseAdmin->enableHFileBackup()). Good to have some closing behavior.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15364>

          How costly is this op; the reason I ask is it will be useful only once; once the znode is there, this call has no value.
          (Still learning zk stuff, kindly ignore if you think so)

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment15375>

          minor nit: "is" enabled?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment15376>

          you made the deleteRegion to rawDeleteRegion? lol, but that sounds a bit secondary
          Why not keep delete or sth deleteWithNoArchive?

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment15369>

          Yes, please add some doc to it.
          It is instantiated in HMaster, so can't be abstract?
          <>Tracker and <>Monitor doesn't seem to give a clearer picture, somehow, especially, when tracker implements the monitor?

          • Himanshu

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6914 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15366 > Can the checks in the while loop be reversed, so that we check the counter first and then call the allRegionsDoneBackup(). Will this help in removing the call to disableHFileBackup, or am i missing anything here. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15363 > yeah, instantiation of this call is done more than once in a single code path (call to HBaseAdmin->enableHFileBackup()). Good to have some closing behavior. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15364 > How costly is this op; the reason I ask is it will be useful only once; once the znode is there, this call has no value. (Still learning zk stuff, kindly ignore if you think so) src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment15375 > minor nit: "is" enabled? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment15376 > you made the deleteRegion to rawDeleteRegion? lol, but that sounds a bit secondary Why not keep delete or sth deleteWithNoArchive? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment15369 > Yes, please add some doc to it. It is instantiated in HMaster, so can't be abstract? <>Tracker and <>Monitor doesn't seem to give a clearer picture, somehow, especially, when tracker implements the monitor? Himanshu On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review6917
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment15382>

          You wouldn't have chance to do that if it fails as there will be an IOException?

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment15385>

          I see; you trying to archive files, and just delete in case there is any failure for some hfiles.

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment15387>

          I miss this; who is calling this addTable?
          It would be great to have the usecase.
          (sorry for reviewing it piece meal, its a huge patch as per my standards )

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          <https://reviews.apache.org/r/4633/#comment15389>

          And, what are the better alternatives (some subclass holding all these constants, or something more aesthetic).

          • Himanshu

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6917 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment15382 > You wouldn't have chance to do that if it fails as there will be an IOException? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment15385 > I see; you trying to archive files, and just delete in case there is any failure for some hfiles. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment15387 > I miss this; who is calling this addTable? It would be great to have the usecase. (sorry for reviewing it piece meal, its a huge patch as per my standards ) src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java < https://reviews.apache.org/r/4633/#comment15389 > And, what are the better alternatives (some subclass holding all these constants, or something more aesthetic). Himanshu On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Jesse Yates added a comment -

          Adding dependency on transactional, recoverable deleteRecursively in ZKUtil. Makes this patch notably (if not significantly) simpler.

          Show
          Jesse Yates added a comment - Adding dependency on transactional, recoverable deleteRecursively in ZKUtil. Makes this patch notably (if not significantly) simpler.
          Hide
          Jesse Yates added a comment -

          Talked offline with Lars about the design of this patch. Currently, it is updating ZK with the table to archive and then essentially waiting on all the RS to respond that they recieved the update (simple barrier) before considering backups enabled (in the syncrhonous case). This means we get high overhead to zk when turning on/off backups, but not a lot when hfiles are deleted. However, this adds a lot of complexity around watching for nodes, making sure you get updates, etc. which has been the source of a bit of pain in the development of this patch.

          The alternative design is one where the client just writes the 'enable archiving on this table' node to ZK and then when any of the RS want to delete an hfile they go an read ZK. The thought here is that we always have a consistent view of which tables need to be archived without the complexity of keeping up-to-date with the latest state. This means a much simpler design for checking if we should archive a table and very fast enable/disable for backups.

          Clearly, there is a tradeoff in that we have to make a network roundtrip to read if we should delete or archive a table, but there is little else that needs to happen on the server or wire (zk holds the info in memory and all thats happening is a checkExists). However, compared to the time cost in deleting a multi-MB to GB file, this roundtrip seems negligable.

          The last alternative would be the dynamic configuration that nicholas recently rolled into trunk. From what I gather from stack's comments, this would take a disable/enable on a table for the change to propagate to the HTD (please correct me if I'm wrong here). This just seems like a lot of flux on a table for a backup situation that I was thinking could occur as often as a nightly basis. Making the database unavailable just for backup is unreasonable and should be done dynamically, with minimal impact to the cluster.

          What does everyone think? I'm happy to do the rewrite on the patch to make either of the alternative designs work, as the current on is not quite ready (just needs to have the barrier stuff reworked slightly to have new/split regions register for archiving).

          Show
          Jesse Yates added a comment - Talked offline with Lars about the design of this patch. Currently, it is updating ZK with the table to archive and then essentially waiting on all the RS to respond that they recieved the update (simple barrier) before considering backups enabled (in the syncrhonous case). This means we get high overhead to zk when turning on/off backups, but not a lot when hfiles are deleted. However, this adds a lot of complexity around watching for nodes, making sure you get updates, etc. which has been the source of a bit of pain in the development of this patch. The alternative design is one where the client just writes the 'enable archiving on this table' node to ZK and then when any of the RS want to delete an hfile they go an read ZK. The thought here is that we always have a consistent view of which tables need to be archived without the complexity of keeping up-to-date with the latest state. This means a much simpler design for checking if we should archive a table and very fast enable/disable for backups. Clearly, there is a tradeoff in that we have to make a network roundtrip to read if we should delete or archive a table, but there is little else that needs to happen on the server or wire (zk holds the info in memory and all thats happening is a checkExists). However, compared to the time cost in deleting a multi-MB to GB file, this roundtrip seems negligable. The last alternative would be the dynamic configuration that nicholas recently rolled into trunk. From what I gather from stack's comments, this would take a disable/enable on a table for the change to propagate to the HTD (please correct me if I'm wrong here). This just seems like a lot of flux on a table for a backup situation that I was thinking could occur as often as a nightly basis. Making the database unavailable just for backup is unreasonable and should be done dynamically, with minimal impact to the cluster. What does everyone think? I'm happy to do the rewrite on the patch to make either of the alternative designs work, as the current on is not quite ready (just needs to have the barrier stuff reworked slightly to have new/split regions register for archiving).
          Hide
          stack added a comment -

          Is the problem in #1 the client waiting on acks from all the regionservers? Does it need to do this? Can it not just set the state up in zk and then just move on (You have this in your patch already if I remember rightly). Do you want the RS's acknowledging that they have been set into backup mode? They could set a flag up in zk but this gets torturous when say we add a new feature that wants to do some thing similar.

          If we had a dynamic Configuration system, one that didn't require roll of table to set the table 'read-only' or 'in-back-up mode', would that help here?

          One option #2, yeah, its a pain going to zk for each WAL when there is this callback mechanism that all RS are subscribed to anyways. For sure could poll zk the first time but should then cache the setting and only drop it later if a callback says it changed.

          Agree roll of table to set the backup flag is much too heavyweight.

          Show
          stack added a comment - Is the problem in #1 the client waiting on acks from all the regionservers? Does it need to do this? Can it not just set the state up in zk and then just move on (You have this in your patch already if I remember rightly). Do you want the RS's acknowledging that they have been set into backup mode? They could set a flag up in zk but this gets torturous when say we add a new feature that wants to do some thing similar. If we had a dynamic Configuration system, one that didn't require roll of table to set the table 'read-only' or 'in-back-up mode', would that help here? One option #2, yeah, its a pain going to zk for each WAL when there is this callback mechanism that all RS are subscribed to anyways. For sure could poll zk the first time but should then cache the setting and only drop it later if a callback says it changed. Agree roll of table to set the backup flag is much too heavyweight.
          Hide
          Ted Yu added a comment -

          Currently we have the following in enable():

              // then add the table to the list of znodes to archive
              String tableNode = getTableNode(zooKeeper, table);
              ZKUtil.createSetData(zooKeeper, tableNode, archive);
          

          If we encode the backup mode in the data associated with the table node, the region servers would be able to register watcher for the table nodes (under zooKeeper.archiveHFileZNode). This way region servers would get notification from zk about whether backup mode is effective.

          Show
          Ted Yu added a comment - Currently we have the following in enable(): // then add the table to the list of znodes to archive String tableNode = getTableNode(zooKeeper, table); ZKUtil.createSetData(zooKeeper, tableNode, archive); If we encode the backup mode in the data associated with the table node, the region servers would be able to register watcher for the table nodes (under zooKeeper.archiveHFileZNode). This way region servers would get notification from zk about whether backup mode is effective.
          Hide
          Jesse Yates added a comment -

          @Ted, stack: there is a race condition where the client thinks that backups are active, when ZK hasn't in fact sent watch notifications to all the RS. Granted, this is a small time window (hopefully), but is definitely a concern and something that I've seen take up to a few seconds to propagate. It's probably not a big deal, but that is the concern and if you are doing tables backups based on when archiving is enabled, you want to make sure archiving is in fact enabled on all nodes.

          If we encode the backup mode in the data associated with the table node, the region servers would be able to register watcher for the table nodes

          Are you basically talking about doing per-table configuration storage in the table znode? I think this would be completely viable with the transaction stuff - essentially check the version and update the data if matches expected, if not propose new value and try again (see below for paxos concerns, though this isn't actually all that rough from the client side).

          If we had a dynamic Configuration system, one that didn't require roll of table to set the table 'read-only' or 'in-back-up mode', would that help here?

          That's essentially what I was building here, but just for state changes. Unfortunately, its pretty close to Paxos (propsing new value, getting acceptance, confirming value update to client), and has a lot of complexity to ensure changes propagate - the latter being the biggest concern from a 'consistent view of the cluster by the client' standpoint.

          For sure could poll zk the first time but should then cache the setting and only drop it later if a callback says it changed.

          If they are disabled, they need to check everytime to see if it has been enabled. Once it has been enabled, it could then wait for the disable to propagate via notifications because the excess backup files aren't a big deal, but the loss of backup files is an issue.

          Show
          Jesse Yates added a comment - @Ted, stack: there is a race condition where the client thinks that backups are active, when ZK hasn't in fact sent watch notifications to all the RS. Granted, this is a small time window (hopefully), but is definitely a concern and something that I've seen take up to a few seconds to propagate. It's probably not a big deal, but that is the concern and if you are doing tables backups based on when archiving is enabled, you want to make sure archiving is in fact enabled on all nodes. If we encode the backup mode in the data associated with the table node, the region servers would be able to register watcher for the table nodes Are you basically talking about doing per-table configuration storage in the table znode? I think this would be completely viable with the transaction stuff - essentially check the version and update the data if matches expected, if not propose new value and try again (see below for paxos concerns, though this isn't actually all that rough from the client side). If we had a dynamic Configuration system, one that didn't require roll of table to set the table 'read-only' or 'in-back-up mode', would that help here? That's essentially what I was building here, but just for state changes. Unfortunately, its pretty close to Paxos (propsing new value, getting acceptance, confirming value update to client), and has a lot of complexity to ensure changes propagate - the latter being the biggest concern from a 'consistent view of the cluster by the client' standpoint. For sure could poll zk the first time but should then cache the setting and only drop it later if a callback says it changed. If they are disabled, they need to check everytime to see if it has been enabled. Once it has been enabled, it could then wait for the disable to propagate via notifications because the excess backup files aren't a big deal, but the loss of backup files is an issue.
          Hide
          stack added a comment -

          ...but is definitely a concern and something that I've seen take up to a few seconds to propagate.

          Yeah. If you don't want a window, query the regionservers (you'll need to add something to query but...)

          ... Are you basically talking about doing per-table configuration storage in the table znode?

          I was stating then that we already are doing a per table attribute up in zk – whether enabled or disabled – and that rather than do up new nodes for a new attribute that instead we should add to the table znode the new attribute. That was then. Now I'm suggesting we put all config up there. We could start w/ HTD if we want to keep it table scoped (we'd have another tier in front of the one Nicolas added, a dynamic one).

          If the above too ambitious, we should at least generalize the table znode so can add attributes and we might as well pb serialize the HTD as anything else?

          ...If they are disabled, they need to check everytime to see if it has been enabled

          Or just watch the table znode and if it changes, check if backup has been flipped on.

          Show
          stack added a comment - ...but is definitely a concern and something that I've seen take up to a few seconds to propagate. Yeah. If you don't want a window, query the regionservers (you'll need to add something to query but...) ... Are you basically talking about doing per-table configuration storage in the table znode? I was stating then that we already are doing a per table attribute up in zk – whether enabled or disabled – and that rather than do up new nodes for a new attribute that instead we should add to the table znode the new attribute. That was then. Now I'm suggesting we put all config up there. We could start w/ HTD if we want to keep it table scoped (we'd have another tier in front of the one Nicolas added, a dynamic one). If the above too ambitious, we should at least generalize the table znode so can add attributes and we might as well pb serialize the HTD as anything else? ...If they are disabled, they need to check everytime to see if it has been enabled Or just watch the table znode and if it changes, check if backup has been flipped on.
          Hide
          Jesse Yates added a comment -

          Yeah. If you don't want a window, query the regionservers (you'll need to add something to query but...)

          I was trying to avoid making a change to the RS interface just for the check to see if it has propagated. However, this is a valid case, but requires a scan of META to see the regions and associated RS that need to be pinged. A little rough on IO, but certainly doable.

          ...If they are disabled, they need to check everytime to see if it has been enabled

          Did this to minimize the latency for changes to propagate and minimize the zk logic for tracking nodes. If we tied the watching of the table znode state/config with updating the status, then yeah, this would be doable. I can see doing another set of property listeners for table properties - could actually be really nice. Wanna file a ticket?

          Show
          Jesse Yates added a comment - Yeah. If you don't want a window, query the regionservers (you'll need to add something to query but...) I was trying to avoid making a change to the RS interface just for the check to see if it has propagated. However, this is a valid case, but requires a scan of META to see the regions and associated RS that need to be pinged. A little rough on IO, but certainly doable. ...If they are disabled, they need to check everytime to see if it has been enabled Did this to minimize the latency for changes to propagate and minimize the zk logic for tracking nodes. If we tied the watching of the table znode state/config with updating the status, then yeah, this would be doable. I can see doing another set of property listeners for table properties - could actually be really nice. Wanna file a ticket?
          Hide
          Lars Hofhansl added a comment -

          What I had in mind when I filed this was something quite simple: A single znode that all RegionServers would check when deleting any HFile and then based on existence or value of that znode either delete the HFile or rename it. Of things are never quite as simple...

          To get a guaranteed consistent snapshot the RegionServers need to check for the znode's value synchronously in the delete path (or at least I see no other way). Otherwise there are times when the RegionServers do not agree and some files will be deleted and some will be backed up with no possibility for the client to know exactly as of when the backup would be consistent.

          Since HFiles are deleted as result of a compaction in an asynchronous thread, synchronously checking the znode should not cause performance issues, unless we fear we'll overload ZK.

          This simple solution would add special code for just this scenario, which is bad. At the same time it would be relatively simple (famous last words), so that's something to weigh.

          Show
          Lars Hofhansl added a comment - What I had in mind when I filed this was something quite simple: A single znode that all RegionServers would check when deleting any HFile and then based on existence or value of that znode either delete the HFile or rename it. Of things are never quite as simple... To get a guaranteed consistent snapshot the RegionServers need to check for the znode's value synchronously in the delete path (or at least I see no other way). Otherwise there are times when the RegionServers do not agree and some files will be deleted and some will be backed up with no possibility for the client to know exactly as of when the backup would be consistent. Since HFiles are deleted as result of a compaction in an asynchronous thread, synchronously checking the znode should not cause performance issues, unless we fear we'll overload ZK. This simple solution would add special code for just this scenario, which is bad. At the same time it would be relatively simple (famous last words), so that's something to weigh.
          Hide
          stack added a comment -

          To get a guaranteed consistent snapshot the RegionServers need to check for the znode's value synchronously in the delete path (or at least I see no other way). Otherwise there are times when the RegionServers do not agree and some files will be deleted and some will be backed up with no possibility for the client to know exactly as of when the backup would be consistent.

          This would make for the narrowest possible window regards whether backup is on or off.

          Does it have to be a custom znode? If we had a Configuration or Table znode, it could read the content? Maybe checking existence is cheaper than reading znode content though?

          Show
          stack added a comment - To get a guaranteed consistent snapshot the RegionServers need to check for the znode's value synchronously in the delete path (or at least I see no other way). Otherwise there are times when the RegionServers do not agree and some files will be deleted and some will be backed up with no possibility for the client to know exactly as of when the backup would be consistent. This would make for the narrowest possible window regards whether backup is on or off. Does it have to be a custom znode? If we had a Configuration or Table znode, it could read the content? Maybe checking existence is cheaper than reading znode content though?
          Hide
          Lars Hofhansl added a comment -

          For this I was actually thinking a single cluster-wide setting ("This HBase cluster is now in backup mode"), which would affect all tables would be sufficient.

          But could do per table as well as Jesse has done and it definitely does not need to be a custom znode, checking a value is fine.

          Show
          Lars Hofhansl added a comment - For this I was actually thinking a single cluster-wide setting ("This HBase cluster is now in backup mode"), which would affect all tables would be sufficient. But could do per table as well as Jesse has done and it definitely does not need to be a custom znode, checking a value is fine.
          Hide
          Jesse Yates added a comment -

          Maybe checking existence is cheaper than reading znode content though?

          I don't think it would be appreciable. It still has to to go memory to check existence, so the only loss is the time to write the content over the wire (negligible in our case when we have around a 1KB of configuration info).

          If we had a Configuration or Table znode, it could read the content

          Certainly. But then there is the work to finish that code (HBASE-3909 if I'm not mistaken), which still seems a long way off.

          This may all (except for the table config stuff, which would be an arguably 'better' version) be moot as I do have a version working now that does synchronous checking to make sure that all necessary RS received the update have updated zk that they have received the notification. Avoids the problem of splits/merges per region as its all on the RS level. There is still a bit of cleanup I need to do on the patch before posting the another review, but the core of the functionality is there.

          Show
          Jesse Yates added a comment - Maybe checking existence is cheaper than reading znode content though? I don't think it would be appreciable. It still has to to go memory to check existence, so the only loss is the time to write the content over the wire (negligible in our case when we have around a 1KB of configuration info). If we had a Configuration or Table znode, it could read the content Certainly. But then there is the work to finish that code ( HBASE-3909 if I'm not mistaken), which still seems a long way off. This may all (except for the table config stuff, which would be an arguably 'better' version) be moot as I do have a version working now that does synchronous checking to make sure that all necessary RS received the update have updated zk that they have received the notification. Avoids the problem of splits/merges per region as its all on the RS level. There is still a bit of cleanup I need to do on the patch before posting the another review, but the core of the functionality is there.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7224
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/HConstants.java
          <https://reviews.apache.org/r/4633/#comment15915>

          Things shared across, IMO, should stay here. Also, the znode style seems to be to put it in constants. Probably a bigger issue around how the zkutil works.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15916>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15921>

          @Himanshu - yeah, that should help a bit.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment15977>

          not anymore - new version ignores specifying an archive dir and instead goes with a cluster wide config.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment15980>

          Done.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment16003>

          done.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment16004>

          this class is more of just a way to wrap this up from the admin. Otherwise, it would just be there.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment16005>

          has to be added, since createWithParents just creates the znode, not the data in that node. Could modify ZKUtil, but this seemed sufficient and don't think we have a lot of use cases where the node has data. If we do, think it would be reasonable to add a zk call.

          In terms of cost, its basically 1 ping to the server to ensure that the node exists, which would have to happen anyways if we create the table node since we can't bundle calls together until we make ZK 3.4 a requirement.

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment16006>

          moved to o.a.h.h.backup package.

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment16007>

          Moved to o.a.h.h.backup, but then still needs to be public.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16008>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16009>

          done.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16010>

          nope as it just uses the passed in zk, rather than creating its own. start here is just a way to bootstrap it to the current state of the system.

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment16011>

          Yeah, but if you have an IOException creating a directory you are kind of hosed anyways.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment16012>

          Turns out this is overly complicated based on the fact that hashes here are incredibly unlikely to conflict, so still doing a timestamp, but this time just appending the ts to the name of the file and moving it, rather than all this business with directories.

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16013>

          gone, in backup (see above).

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment16014>

          yeah, we are collecting them here, but since its used across all, moved them to their own package.

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java
          <https://reviews.apache.org/r/4633/#comment16015>

          its used lower, so RS can hook in confirming joining archiving.

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
          <https://reviews.apache.org/r/4633/#comment16016>

          agreed. Someone should think about it some more and file a jira. Maybe some configuration based loading of short-lived visitor-like classes?

          • Jesse

          On 2012-04-07 19:51:11, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-07 19:51:11)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd

          src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml 44ee689

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7224 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/HConstants.java < https://reviews.apache.org/r/4633/#comment15915 > Things shared across, IMO, should stay here. Also, the znode style seems to be to put it in constants. Probably a bigger issue around how the zkutil works. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15916 > done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15921 > @Himanshu - yeah, that should help a bit. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment15977 > not anymore - new version ignores specifying an archive dir and instead goes with a cluster wide config. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment15980 > Done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment16003 > done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment16004 > this class is more of just a way to wrap this up from the admin. Otherwise, it would just be there. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment16005 > has to be added, since createWithParents just creates the znode, not the data in that node. Could modify ZKUtil, but this seemed sufficient and don't think we have a lot of use cases where the node has data. If we do, think it would be reasonable to add a zk call. In terms of cost, its basically 1 ping to the server to ensure that the node exists, which would have to happen anyways if we create the table node since we can't bundle calls together until we make ZK 3.4 a requirement. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment16006 > moved to o.a.h.h.backup package. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment16007 > Moved to o.a.h.h.backup, but then still needs to be public. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16008 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16009 > done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16010 > nope as it just uses the passed in zk, rather than creating its own. start here is just a way to bootstrap it to the current state of the system. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment16011 > Yeah, but if you have an IOException creating a directory you are kind of hosed anyways. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment16012 > Turns out this is overly complicated based on the fact that hashes here are incredibly unlikely to conflict, so still doing a timestamp, but this time just appending the ts to the name of the file and moving it, rather than all this business with directories. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16013 > gone, in backup (see above). src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment16014 > yeah, we are collecting them here, but since its used across all, moved them to their own package. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java < https://reviews.apache.org/r/4633/#comment16015 > its used lower, so RS can hook in confirming joining archiving. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java < https://reviews.apache.org/r/4633/#comment16016 > agreed. Someone should think about it some more and file a jira. Maybe some configuration based loading of short-lived visitor-like classes? Jesse On 2012-04-07 19:51:11, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-07 19:51:11) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/
          -----------------------------------------------------------

          (Updated 2012-04-26 00:24:36.588824)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Changes
          -------

          Updates:

          • check-in for received archive request is synchronous (no change there) per regionserver
          • configuration based rather than request based archive directory
          • extracting out tabletracker from the archivetracker
          • stop() method (and usage) on ArchiveManager
          • simplified timestamped backups
          • cleanup of archive files is now done by external utility, rather than attempting to deal with master failure and recovery

          Summary
          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:
          Updated the HMasterInterface to pass the calls onto the zookeeper.
          Added a zk listener to handle updates from the master to the RS to backup.
          Added a utility for removing files and finding archive directories
          Added tests for the regionserver and catalogjanitor approaches.
          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.
          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72
          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53
          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f
          src/main/resources/hbase-default.xml f54b345
          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759
          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing
          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-26 00:24:36.588824) Review request for hbase, Michael Stack and Lars Hofhansl. Changes ------- Updates: check-in for received archive request is synchronous (no change there) per regionserver configuration based rather than request based archive directory extracting out tabletracker from the archivetracker stop() method (and usage) on ArchiveManager simplified timestamped backups cleanup of archive files is now done by external utility, rather than attempting to deal with master failure and recovery Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs (updated) src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7318
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16155>

          'for which' -> 'the'

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16156>

          Insert 'whose' between 'tables' and 'HFiles'

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16157>

          The condition for calling addAndReWatchTable() is when table exists.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16159>

          The method name is confusing.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16158>

          When would this be true, considering that we have come through line 106 ?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16160>

          'reup' -> 'renew' ?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16161>

          getTracker() and direct read of tracker appear on the same line ?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16162>

          Can you use archivedTables.addAll() ?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment16163>

          Remove the 's' following 'Region'

          • Ted

          On 2012-04-26 00:24:36, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-26 00:24:36)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml f54b345

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7318 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16155 > 'for which' -> 'the' src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16156 > Insert 'whose' between 'tables' and 'HFiles' src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16157 > The condition for calling addAndReWatchTable() is when table exists. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16159 > The method name is confusing. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16158 > When would this be true, considering that we have come through line 106 ? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16160 > 'reup' -> 'renew' ? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16161 > getTracker() and direct read of tracker appear on the same line ? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16162 > Can you use archivedTables.addAll() ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment16163 > Remove the 's' following 'Region' Ted On 2012-04-26 00:24:36, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-26 00:24:36) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7410
          -----------------------------------------------------------

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16338>

          agreed. I'll try to find something better.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16339>

          zk add/remove conditions where you need to keep re-adding watches. figure we do the check for the table at the same time.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16340>

          removed the creation of the tracker in favor of just passing in one - its really light weight and this should remove some of the issues around it; in short, yes.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16341>

          switching this mechanism to allow more fine grained listening by table trackers.

          • Jesse

          On 2012-04-26 00:24:36, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-04-26 00:24:36)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml f54b345

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7410 ----------------------------------------------------------- src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16338 > agreed. I'll try to find something better. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16339 > zk add/remove conditions where you need to keep re-adding watches. figure we do the check for the table at the same time. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16340 > removed the creation of the tracker in favor of just passing in one - its really light weight and this should remove some of the issues around it; in short, yes. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16341 > switching this mechanism to allow more fine grained listening by table trackers. Jesse On 2012-04-26 00:24:36, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-04-26 00:24:36) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/
          -----------------------------------------------------------

          (Updated 2012-05-01 02:42:34.215505)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Changes
          -------

          • TableTracker is now an abstract class that the catalogjanitor and rs are
            both using to register for tracking a table.
          • Adding monitoring in admin to check for cj to join archiving (before was missed) and including test
          • Slight cleanup from Ted's comments.

          Summary
          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:
          Updated the HMasterInterface to pass the calls onto the zookeeper.
          Added a zk listener to handle updates from the master to the RS to backup.
          Added a utility for removing files and finding archive directories
          Added tests for the regionserver and catalogjanitor approaches.
          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.
          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72
          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53
          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f
          src/main/resources/hbase-default.xml f54b345
          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759
          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing
          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-05-01 02:42:34.215505) Review request for hbase, Michael Stack and Lars Hofhansl. Changes ------- TableTracker is now an abstract class that the catalogjanitor and rs are both using to register for tracking a table. Adding monitoring in admin to check for cj to join archiving (before was missed) and including test Slight cleanup from Ted's comments. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs (updated) src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          @Jesse:
          Do you want to attach patch so that Hadoop QA can run test suite ?

          Show
          Ted Yu added a comment - @Jesse: Do you want to attach patch so that Hadoop QA can run test suite ?
          Hide
          Jesse Yates added a comment -

          Attaching patch rebased on trunk (but otherwise same as RB) as per request.

          Show
          Jesse Yates added a comment - Attaching patch rebased on trunk (but otherwise same as RB) as per request.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525163/java_HBASE-5547_v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestCheckTestClasses

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525163/java_HBASE-5547_v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestCheckTestClasses Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          Attaching version with fixed test and correction to the one real findbugs warning.

          Show
          Jesse Yates added a comment - Attaching version with fixed test and correction to the one real findbugs warning.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525196/java_HBASE-5547_v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestRegionHFileArchiving
          org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525196/java_HBASE-5547_v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestRegionHFileArchiving org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          arg! Thought I had resolved the concurrency issue in TestRegionHFileArchiving. Looking into it.

          Show
          Jesse Yates added a comment - arg! Thought I had resolved the concurrency issue in TestRegionHFileArchiving. Looking into it.
          Hide
          Jesse Yates added a comment -

          Attaching new patch. Just ran the test 10 times locally, no failures.

          Show
          Jesse Yates added a comment - Attaching new patch. Just ran the test 10 times locally, no failures.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7448
          -----------------------------------------------------------

          Great work. High-level, do we have to have backup/archiving touch all parts of the code base? Can we not hide it behind more general Interfaces at least when we get down into Store and Region classes

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment16389>

          Tables or hfiles?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment16384>

          Should this interface be at top level? Its used out of the master and regionserver packages

          The implementations could be stay under backup long as we only use the Interface?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16390>

          hfiles or tables?

          Looks like this class would select hfiles by selected tables

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16387>

          Tracker in hbase usually means tracking znodes up in zookeeper. This tracks something else. Could confuse. Use different action? Manager

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16385>

          Good. I like that you are taking a 'Server' here rather than a Master or Regionserver or even a ServerName.

          Does this have to be public? Seems like its utility used by the later HFileArchiveTracker.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16386>

          Do we have to have a set? Does this Set change during live of this tracker?

          Should javadoc say that previous members of Set are cleared out?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16388>

          good

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16392>

          How does this tracker relate to the containing class? They are different types of trackers? Could confuse?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16391>

          Why pass it in? Why not just create one and keep it running internally?

          The class HFileArchiveTableTracker is a little confusing....

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16395>

          Want to mention in class comment that user needs to call start, etc., to start up the tracking?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16399>

          I think it kinda bad that zkw does this... you'd think each tracker would look after its own stuff (this is not you.. you are folllowing the established pattern as you should – I'm just moaning)

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16400>

          Is this what ZKUtil.getNodeName does?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16401>

          nit style. if you did if (!path.startsWith...) return; ... you could save an indent of the whole method. Next time.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16403>

          These znodes have nothing in them... just the tablename of the znode is used? Nothing to pb then?

          src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16405>

          This is under each table znode? Why we inherit from HFileArchiveTableTracker?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment16407>

          Is this doing backup? Or making it so the hfiles are not deleted on this table?

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment16409>

          ditto... this just throws a switch on the don't delete operation? Something else does the backup?

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment16410>

          table hfiles?

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment16411>

          Why this class have to have both these classes? Can it not just make do w/ Interface? If not, does Interface need broadening?

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment16412>

          Should this be getHFileManager?

          Should it be getTableArchivingManager?

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16413>

          Can this not just be Interface?

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16414>

          You think this an hfile archive tracker or is it a table archive tracker?

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16415>

          Does the master need to know anythign about this? Can this be done in the catalogjanitor completely?

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16416>

          Is the testing code in same package? IF so, wouldn't need the public qualifier

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16417>

          ditto

          ... this doesn't exist already?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16418>

          Interface? Odd that we have a reference to this in here.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16419>

          Could this be a different, more abstract Interface? HFileArchiver with an 'archive' method on it which return true if we are to archive otherwise, false if its for delete.

          Would like to avoid too much of the backup detail coming down deep in here into HRegion.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16420>

          Or, could the whole delete/archive thing be hidden behind an hfile disposer Interface? Pass it hfiles to dispose of. It will delete or archive?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16421>

          This archiving code shouldn't be in here? Should be in its own class?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16422>

          Yeah, seems odd polluting hregion w/ archiving?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16423>

          Can this be Interface?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16424>

          A factory? Is that too much?

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16425>

          You should leave these. These are ok by our style convention.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16426>

          Yeah, these are ok... don't change them.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16427>

          This for tests too?

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          <https://reviews.apache.org/r/4633/#comment16428>

          Does this have to come out here?

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment16429>

          Has to come down in here? Can't this be done at higher up levels?

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment16430>

          Ditto on archiving/delete pollution down in here.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment16431>

          These lines have to be this long?

          Does this tool belong in backup package?

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment16432>

          Has to reach over into these packages?

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16433>

          This has to be in here?

          • Michael

          On 2012-05-01 02:42:34, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-05-01 02:42:34)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml f54b345

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7448 ----------------------------------------------------------- Great work. High-level, do we have to have backup/archiving touch all parts of the code base? Can we not hide it behind more general Interfaces at least when we get down into Store and Region classes src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment16389 > Tables or hfiles? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment16384 > Should this interface be at top level? Its used out of the master and regionserver packages The implementations could be stay under backup long as we only use the Interface? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16390 > hfiles or tables? Looks like this class would select hfiles by selected tables src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16387 > Tracker in hbase usually means tracking znodes up in zookeeper. This tracks something else. Could confuse. Use different action? Manager src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16385 > Good. I like that you are taking a 'Server' here rather than a Master or Regionserver or even a ServerName. Does this have to be public? Seems like its utility used by the later HFileArchiveTracker. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16386 > Do we have to have a set? Does this Set change during live of this tracker? Should javadoc say that previous members of Set are cleared out? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16388 > good src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16392 > How does this tracker relate to the containing class? They are different types of trackers? Could confuse? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16391 > Why pass it in? Why not just create one and keep it running internally? The class HFileArchiveTableTracker is a little confusing.... src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16395 > Want to mention in class comment that user needs to call start, etc., to start up the tracking? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16399 > I think it kinda bad that zkw does this... you'd think each tracker would look after its own stuff (this is not you.. you are folllowing the established pattern as you should – I'm just moaning) src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16400 > Is this what ZKUtil.getNodeName does? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16401 > nit style. if you did if (!path.startsWith...) return; ... you could save an indent of the whole method. Next time. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16403 > These znodes have nothing in them... just the tablename of the znode is used? Nothing to pb then? src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16405 > This is under each table znode? Why we inherit from HFileArchiveTableTracker? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment16407 > Is this doing backup? Or making it so the hfiles are not deleted on this table? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment16409 > ditto... this just throws a switch on the don't delete operation? Something else does the backup? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment16410 > table hfiles? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment16411 > Why this class have to have both these classes? Can it not just make do w/ Interface? If not, does Interface need broadening? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment16412 > Should this be getHFileManager? Should it be getTableArchivingManager? src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16413 > Can this not just be Interface? src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16414 > You think this an hfile archive tracker or is it a table archive tracker? src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16415 > Does the master need to know anythign about this? Can this be done in the catalogjanitor completely? src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16416 > Is the testing code in same package? IF so, wouldn't need the public qualifier src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16417 > ditto ... this doesn't exist already? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16418 > Interface? Odd that we have a reference to this in here. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16419 > Could this be a different, more abstract Interface? HFileArchiver with an 'archive' method on it which return true if we are to archive otherwise, false if its for delete. Would like to avoid too much of the backup detail coming down deep in here into HRegion. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16420 > Or, could the whole delete/archive thing be hidden behind an hfile disposer Interface? Pass it hfiles to dispose of. It will delete or archive? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16421 > This archiving code shouldn't be in here? Should be in its own class? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16422 > Yeah, seems odd polluting hregion w/ archiving? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16423 > Can this be Interface? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16424 > A factory? Is that too much? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16425 > You should leave these. These are ok by our style convention. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16426 > Yeah, these are ok... don't change them. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16427 > This for tests too? src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java < https://reviews.apache.org/r/4633/#comment16428 > Does this have to come out here? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment16429 > Has to come down in here? Can't this be done at higher up levels? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment16430 > Ditto on archiving/delete pollution down in here. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment16431 > These lines have to be this long? Does this tool belong in backup package? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment16432 > Has to reach over into these packages? src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java < https://reviews.apache.org/r/4633/#comment16433 > This has to be in here? Michael On 2012-05-01 02:42:34, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-05-01 02:42:34) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7458
          -----------------------------------------------------------

          Thanks for the review. Going to do a bit of investigation as to more commonality/interface hiding, but unfortunately we handle hfiles in both the RS and Master (and need to actuate from the client), so we need to touch a lot.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment16469>

          hfiles, tables are another patch

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java
          <https://reviews.apache.org/r/4633/#comment16468>

          top-level as in o.a.h.hbase? Seems like a bit of clutter and this is just a specialization on that packaging and IMO doesn't preclude either master or rs packages from usage.

          But I can move it if there are strong feelings.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16470>

          your right, fubbing my commenting a bit.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16471>

          But its does tracking! And it doesn't actually actuate any changes on the rest of the cluster so Manager seems bad too. How about Monitor?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16472>

          Started thinking about your later comment about a factory in HRS.java and think I'm going with the static factory method there. This still needs to be public for testing, but in general we will use the the ServerArchiveTableTracker (and I'll add docs to that effect as well).

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java
          <https://reviews.apache.org/r/4633/#comment16473>

          It won't be used except for at startup, but seemed bad practice to prematurely read ZK before we get the start call on the parent. At the very least we can say that we are removing the previous tables though.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16474>

          Should be fixed with the naming change above.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16475>

          Yeah, its definitely a bit odd. Going with a factory here and only having the pass in option for testing exposure. See above (tabletracker) and below (hrs) for more info on the factory stuff.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16476>

          done.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16477>

          Yeah, its pretty gross. I've been thinking about the ways we can fix this/overhaul the zookeeper infrastructure.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16478>

          This is actually a little nicer in that it doesn't preclude you from having a table named the same thing as the archive directory.

          Otherwise, you are checking the node vs. the node name of watcher.archiveHFileZNode, which could be the same.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16479>

          Done. Seemed to be the prevailing style, but this goes back to the fact that the watcher stuff if crufty.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16480>

          correct! Though anything in them would naturally be pb-ed

          src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java
          <https://reviews.apache.org/r/4633/#comment16481>

          Inherit from tabletracker so we know when we should archive. However, seems like a bit cleaner to just let the subclasses just override registerTable() for testing.

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment16482>

          what do you mean by backup? Its moving them to another directory, rather than being deleted. Sounds like backup to me

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          <https://reviews.apache.org/r/4633/#comment16483>

          just throws a switch. should note its async, so you could have some extra backup files, but ZK is fast, so it shouldn' be that many.

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java
          <https://reviews.apache.org/r/4633/#comment16484>

          yup.

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment16485>

          Only need one - the other is going to be unused in revised version.

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          <https://reviews.apache.org/r/4633/#comment16486>

          Cruft naming - fixed.

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16487>

          cruft - just needs the tracker now. Paradigm seems to be to not have that be an interface.

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16488>

          per-table hfile archiving tracker. So... both and neither? Name seems to be ok...maybe something like TableHFileArchiveTracker?

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16490>

          master didn't, but didn't want to add all the zk dependencies into the CJ. And if the archive stuff gets more complex, it might need to be in the master (weak excuse, but still...)

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16489>

          truth.

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          <https://reviews.apache.org/r/4633/#comment16491>

          nope, and its a private field.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16492>

          Not really - it only cares about if the HFiles should be archived or not, which is what the monitor is doing.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16493>

          The monitor is what is keeping track of the archive/not logic, but yeah, this could be encapsulated in another class. Wanted to avoid all the redirection, since the code doesn't seem all that complex but moving it out could be reasonable.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <https://reviews.apache.org/r/4633/#comment16494>

          see above.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16495>

          much cleaned up.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16466>

          I was actually initially considering using a factory here, but its definitely not. Rather, just creating the ServerHFileArchiveTracker with a "Server" instance that will give it the correct server name to register, so its kinda factory-ish, but there is only ever one instance (rather than a tabletracker creating class) and is super light weight, as opposed to all the double-checked locking, special creating etc that goes around using a factory in this situation.

          The current impl seems to be the simplest, though I guess we could use an ArchiveTracker factory that takes a zk and Server; seems a bit overkill, though it does hide some of the impl details...

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16467>

          fat-fingered the save on HRS.java. Sorry for the cruft. I'll fix it in the next review.

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16496>

          No, this is for how the delete method is currently setup - takes a regionServerServices, which hides impl details, meaning we need to have this method. But its nice for testing too.

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
          <https://reviews.apache.org/r/4633/#comment16497>

          This ends up being cleaner - otherwise each region actually needs to keep around a reference to the ArchiveMonitor and then pass it into deleteRegion (in HRegion.merge). But if the predominant opinion is to keep that around instead, I can make the change.

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <https://reviews.apache.org/r/4633/#comment16498>

          This is situation 2 where hfiles need to be archived. Tried to keep as much in common between the two situations, but there is limited commonality between the "Archive everything here" and "just archive these couple files" functionality.

          And either case, you need to know if you keep around the hfiles.

          I'll try to do some more combining, but we'll see how much more I can pull out.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment16499>

          Nope, eclipse just didn't want to format this to wrap. Fixed. And moved into backup package.

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          <https://reviews.apache.org/r/4633/#comment16500>

          Just for static methods. Makes it easier for the method interfaces, but I'll look into if simplifying makes more sense.

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
          <https://reviews.apache.org/r/4633/#comment16501>

          regionserverservices - yup. see above comment and see if you think differently.

          • Jesse

          On 2012-05-01 02:42:34, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-05-01 02:42:34)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f

          src/main/resources/hbase-default.xml f54b345

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759

          src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7458 ----------------------------------------------------------- Thanks for the review. Going to do a bit of investigation as to more commonality/interface hiding, but unfortunately we handle hfiles in both the RS and Master (and need to actuate from the client), so we need to touch a lot. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment16469 > hfiles, tables are another patch src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java < https://reviews.apache.org/r/4633/#comment16468 > top-level as in o.a.h.hbase? Seems like a bit of clutter and this is just a specialization on that packaging and IMO doesn't preclude either master or rs packages from usage. But I can move it if there are strong feelings. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16470 > your right, fubbing my commenting a bit. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16471 > But its does tracking! And it doesn't actually actuate any changes on the rest of the cluster so Manager seems bad too. How about Monitor? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16472 > Started thinking about your later comment about a factory in HRS.java and think I'm going with the static factory method there. This still needs to be public for testing, but in general we will use the the ServerArchiveTableTracker (and I'll add docs to that effect as well). src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java < https://reviews.apache.org/r/4633/#comment16473 > It won't be used except for at startup, but seemed bad practice to prematurely read ZK before we get the start call on the parent. At the very least we can say that we are removing the previous tables though. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16474 > Should be fixed with the naming change above. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16475 > Yeah, its definitely a bit odd. Going with a factory here and only having the pass in option for testing exposure. See above (tabletracker) and below (hrs) for more info on the factory stuff. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16476 > done. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16477 > Yeah, its pretty gross. I've been thinking about the ways we can fix this/overhaul the zookeeper infrastructure. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16478 > This is actually a little nicer in that it doesn't preclude you from having a table named the same thing as the archive directory. Otherwise, you are checking the node vs. the node name of watcher.archiveHFileZNode, which could be the same. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16479 > Done. Seemed to be the prevailing style, but this goes back to the fact that the watcher stuff if crufty. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16480 > correct! Though anything in them would naturally be pb-ed src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java < https://reviews.apache.org/r/4633/#comment16481 > Inherit from tabletracker so we know when we should archive. However, seems like a bit cleaner to just let the subclasses just override registerTable() for testing. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment16482 > what do you mean by backup? Its moving them to another directory, rather than being deleted. Sounds like backup to me src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java < https://reviews.apache.org/r/4633/#comment16483 > just throws a switch. should note its async, so you could have some extra backup files, but ZK is fast, so it shouldn' be that many. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java < https://reviews.apache.org/r/4633/#comment16484 > yup. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment16485 > Only need one - the other is going to be unused in revised version. src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java < https://reviews.apache.org/r/4633/#comment16486 > Cruft naming - fixed. src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16487 > cruft - just needs the tracker now. Paradigm seems to be to not have that be an interface. src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16488 > per-table hfile archiving tracker. So... both and neither? Name seems to be ok...maybe something like TableHFileArchiveTracker? src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16490 > master didn't, but didn't want to add all the zk dependencies into the CJ. And if the archive stuff gets more complex, it might need to be in the master (weak excuse, but still...) src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16489 > truth. src/main/java/org/apache/hadoop/hbase/master/HMaster.java < https://reviews.apache.org/r/4633/#comment16491 > nope, and its a private field. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16492 > Not really - it only cares about if the HFiles should be archived or not, which is what the monitor is doing. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16493 > The monitor is what is keeping track of the archive/not logic, but yeah, this could be encapsulated in another class. Wanted to avoid all the redirection, since the code doesn't seem all that complex but moving it out could be reasonable. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < https://reviews.apache.org/r/4633/#comment16494 > see above. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16495 > much cleaned up. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16466 > I was actually initially considering using a factory here, but its definitely not. Rather, just creating the ServerHFileArchiveTracker with a "Server" instance that will give it the correct server name to register, so its kinda factory-ish, but there is only ever one instance (rather than a tabletracker creating class) and is super light weight, as opposed to all the double-checked locking, special creating etc that goes around using a factory in this situation. The current impl seems to be the simplest, though I guess we could use an ArchiveTracker factory that takes a zk and Server; seems a bit overkill, though it does hide some of the impl details... src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16467 > fat-fingered the save on HRS.java. Sorry for the cruft. I'll fix it in the next review. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java < https://reviews.apache.org/r/4633/#comment16496 > No, this is for how the delete method is currently setup - takes a regionServerServices, which hides impl details, meaning we need to have this method. But its nice for testing too. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java < https://reviews.apache.org/r/4633/#comment16497 > This ends up being cleaner - otherwise each region actually needs to keep around a reference to the ArchiveMonitor and then pass it into deleteRegion (in HRegion.merge). But if the predominant opinion is to keep that around instead, I can make the change. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < https://reviews.apache.org/r/4633/#comment16498 > This is situation 2 where hfiles need to be archived. Tried to keep as much in common between the two situations, but there is limited commonality between the "Archive everything here" and "just archive these couple files" functionality. And either case, you need to know if you keep around the hfiles. I'll try to do some more combining, but we'll see how much more I can pull out. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment16499 > Nope, eclipse just didn't want to format this to wrap. Fixed. And moved into backup package. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java < https://reviews.apache.org/r/4633/#comment16500 > Just for static methods. Makes it easier for the method interfaces, but I'll look into if simplifying makes more sense. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java < https://reviews.apache.org/r/4633/#comment16501 > regionserverservices - yup. see above comment and see if you think differently. Jesse On 2012-05-01 02:42:34, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-05-01 02:42:34) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java d47b83a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7858846 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java ea12da4 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml f54b345 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 src/test/java/org/apache/hadoop/hbase/util/TestHFileArchivingCleanup.java PRE-CREATION Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/
          -----------------------------------------------------------

          (Updated 2012-05-10 20:28:50.691971)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Changes
          -------

          Refactoring as per stack's comments. Also, updated the testing to avoid flapping.

          I think this is pretty close to ready for commit.

          Summary
          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:
          Updated the HMasterInterface to pass the calls onto the zookeeper.
          Added a zk listener to handle updates from the master to the RS to backup.
          Added a utility for removing files and finding archive directories
          Added tests for the regionserver and catalogjanitor approaches.
          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.
          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs (updated)


          src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f
          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a
          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e
          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0
          src/main/resources/hbase-default.xml 42d1c4e
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339
          src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 3f61cfb

          Diff: https://reviews.apache.org/r/4633/diff

          Testing
          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-05-10 20:28:50.691971) Review request for hbase, Michael Stack and Lars Hofhansl. Changes ------- Refactoring as per stack's comments. Also, updated the testing to avoid flapping. I think this is pretty close to ready for commit. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs (updated) src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0 src/main/resources/hbase-default.xml 42d1c4e src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339 src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 3f61cfb Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Ted Yu added a comment -

          @Jesse:
          Do you want to post latest patch here for Hadoop QA ?

          Show
          Ted Yu added a comment - @Jesse: Do you want to post latest patch here for Hadoop QA ?
          Hide
          Jesse Yates added a comment -

          Attaching latest version for a QA run.

          Show
          Jesse Yates added a comment - Attaching latest version for a QA run.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4633/#review7794
          -----------------------------------------------------------

          Heading to a meeting.
          Below is first part of review.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment17105>

          We know all is true when table is null, right ?

          Seems we don't need this boolean.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment17100>

          "aren't" -> "are"

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment17101>

          Either remove 'down' or move it to the end of the sentence

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment17102>

          Please check return value.

          I think we should remember the files which we are unable to delete and display them at the end.

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java
          <https://reviews.apache.org/r/4633/#comment17104>

          Would anyone specify start == end ?

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java
          <https://reviews.apache.org/r/4633/#comment17106>

          License, please.

          src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
          <https://reviews.apache.org/r/4633/#comment17107>

          License.

          src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
          <https://reviews.apache.org/r/4633/#comment17108>

          This condition deserves an exception, I think.

          src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java
          <https://reviews.apache.org/r/4633/#comment17109>

          Is it possible to provide more information in this exception ?
          e.g. by using toArchive collection ?

          • Ted

          On 2012-05-10 20:28:50, Jesse Yates wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4633/

          -----------------------------------------------------------

          (Updated 2012-05-10 20:28:50)

          Review request for hbase, Michael Stack and Lars Hofhansl.

          Summary

          -------

          Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap.

          Implementation wise:

          Updated the HMasterInterface to pass the calls onto the zookeeper.

          Added a zk listener to handle updates from the master to the RS to backup.

          Added a utility for removing files and finding archive directories

          Added tests for the regionserver and catalogjanitor approaches.

          Added creation of manager in regionserver.

          This addresses bug HBASE-5547.

          https://issues.apache.org/jira/browse/HBASE-5547

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f

          src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a

          src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e

          src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0

          src/main/resources/hbase-default.xml 42d1c4e

          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339

          src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 3f61cfb

          Diff: https://reviews.apache.org/r/4633/diff

          Testing

          -------

          Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper.

          Thanks,

          Jesse

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7794 ----------------------------------------------------------- Heading to a meeting. Below is first part of review. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment17105 > We know all is true when table is null, right ? Seems we don't need this boolean. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment17100 > "aren't" -> "are" src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment17101 > Either remove 'down' or move it to the end of the sentence src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment17102 > Please check return value. I think we should remember the files which we are unable to delete and display them at the end. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java < https://reviews.apache.org/r/4633/#comment17104 > Would anyone specify start == end ? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java < https://reviews.apache.org/r/4633/#comment17106 > License, please. src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java < https://reviews.apache.org/r/4633/#comment17107 > License. src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java < https://reviews.apache.org/r/4633/#comment17108 > This condition deserves an exception, I think. src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java < https://reviews.apache.org/r/4633/#comment17109 > Is it possible to provide more information in this exception ? e.g. by using toArchive collection ? Ted On 2012-05-10 20:28:50, Jesse Yates wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ ----------------------------------------------------------- (Updated 2012-05-10 20:28:50) Review request for hbase, Michael Stack and Lars Hofhansl. Summary ------- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547 . https://issues.apache.org/jira/browse/HBASE-5547 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java 2f432c4 src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveCleanup.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/HFileDisposer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/backup/TableHFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java e751f65 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1f09be1 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f7ac81a src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 33bc1d0 src/main/resources/hbase-default.xml 42d1c4e src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 9fba339 src/test/java/org/apache/hadoop/hbase/backup/SimpleHFileArchiveTableMonitor.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchivingCleanup.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/backup/TestRegionDisposer.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 3f61cfb Diff: https://reviews.apache.org/r/4633/diff Testing ------- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526422/java_HBASE-5547_v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestDrainingServer

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526422/java_HBASE-5547_v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestDrainingServer Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1840//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          test flapped locally for me, but got it to pass. Is this a known flapper?

          Show
          Jesse Yates added a comment - test flapped locally for me, but got it to pass. Is this a known flapper?
          Hide
          Lars Hofhansl added a comment -

          Sorry for bringing this up again (Jesse you can slap me in the office tomorrow )... But how is this different from a read/write lock scenario?

          I.e. A region server trying to delete an HFile could try obtain the equivalent of a read lock on the table, if that fails it would just rename the file instead of deleting it. A client setting the cluster/table into backup would request the equivalent of a write lock.

          I might be missing something fundamentally, but if we cannot solve this problem with a simple ZK construct (a.k.a. a read/write lock recipe), what is ZK good for?

          Show
          Lars Hofhansl added a comment - Sorry for bringing this up again (Jesse you can slap me in the office tomorrow )... But how is this different from a read/write lock scenario? I.e. A region server trying to delete an HFile could try obtain the equivalent of a read lock on the table, if that fails it would just rename the file instead of deleting it. A client setting the cluster/table into backup would request the equivalent of a write lock. I might be missing something fundamentally, but if we cannot solve this problem with a simple ZK construct (a.k.a. a read/write lock recipe), what is ZK good for?
          Hide
          Jesse Yates added a comment -

          Initially I was against chaining the impl on the basis of latency and RPC overhead. However, after thinking through the situation I think Lars has a point (tip of my hat).

          From the client's perspective, they only need to put a write-lock into /hbase/backup/[table]/. In the async case, the client it done - at some point in the future the write lock will take effect and files for that table will be backed up, rather than deleted.

          In the synchronous case, you can either have not files being deleted (in which case it returns immediately) or you have a bunch of read lock from the servers (more detail on this later), and you have to wait for those read-locks to complete before returning as obtaining a write lock. Note that you do immediately write to zk, but conceptually you just read zk status a bunch of times until the write lock comes into effect - in reality we will use a watch on the oldest child).

          From the server perspective, when attempting to delete an hfile, they first check to see if they have a read lock. If they do, then they go ahead and delete the file. There can be multiple read-locks on a table, but they are ephemeral so if a server dies their read-lock releases. If a write-lock is taken, the server is going to be notified and will remove it's read-lock when its done deleting (a little work needs to be done here, but minimal).

          In terms of the general, read/write lock idea it would have to be almost a full implementation of reentrant-read-write lock (only features we wouldn't need are the write acquiring the read lock and iteratively acquiring locks). This does raise the question of if we want to pull in Netflix's Curator library (https://github.com/Netflix/curator) since it already has an implementation of this functionality, as well as a bunch of other stuff we could use in the future. I had heard that it was a bit buggy, but haven't actually used it (or looked extensively at the code, though the wiki looks great) so I can't weigh in either way in that regard - it does seem a little messy, but not terrible at first blush.

          Initially, this is going to be different than HBASE-5494, though similar, and we could possibly combine the implementations given some more advanced implementation around that ticket (adding per-table functionality (so it becomes /hbase/tablelocks/[table]/function), completion of the goals in HBASE-5991, and fairness in lock-acquirement). Most of this implementation goes away if we want to pull in Curator, with a light layer for HBase.

          Sorry, for the long comment, just wanted to fully flush it out so it can be talked out.

          Show
          Jesse Yates added a comment - Initially I was against chaining the impl on the basis of latency and RPC overhead. However, after thinking through the situation I think Lars has a point ( tip of my hat ). From the client's perspective, they only need to put a write-lock into /hbase/backup/ [table] /. In the async case, the client it done - at some point in the future the write lock will take effect and files for that table will be backed up, rather than deleted. In the synchronous case, you can either have not files being deleted (in which case it returns immediately) or you have a bunch of read lock from the servers (more detail on this later), and you have to wait for those read-locks to complete before returning as obtaining a write lock. Note that you do immediately write to zk, but conceptually you just read zk status a bunch of times until the write lock comes into effect - in reality we will use a watch on the oldest child). From the server perspective, when attempting to delete an hfile, they first check to see if they have a read lock. If they do, then they go ahead and delete the file. There can be multiple read-locks on a table, but they are ephemeral so if a server dies their read-lock releases. If a write-lock is taken, the server is going to be notified and will remove it's read-lock when its done deleting (a little work needs to be done here, but minimal). In terms of the general, read/write lock idea it would have to be almost a full implementation of reentrant-read-write lock (only features we wouldn't need are the write acquiring the read lock and iteratively acquiring locks). This does raise the question of if we want to pull in Netflix's Curator library ( https://github.com/Netflix/curator ) since it already has an implementation of this functionality, as well as a bunch of other stuff we could use in the future. I had heard that it was a bit buggy, but haven't actually used it (or looked extensively at the code, though the wiki looks great) so I can't weigh in either way in that regard - it does seem a little messy, but not terrible at first blush. Initially, this is going to be different than HBASE-5494 , though similar, and we could possibly combine the implementations given some more advanced implementation around that ticket (adding per-table functionality (so it becomes /hbase/tablelocks/ [table] /function), completion of the goals in HBASE-5991 , and fairness in lock-acquirement). Most of this implementation goes away if we want to pull in Curator, with a light layer for HBase. Sorry, for the long comment, just wanted to fully flush it out so it can be talked out.
          Hide
          Ted Yu added a comment -

          Looks like this JIRA would depend on HBASE-5991.

          Show
          Ted Yu added a comment - Looks like this JIRA would depend on HBASE-5991 .
          Hide
          Jesse Yates added a comment -

          HBASE-5991 would be nice, but not necessary. Excited for it to go in though.

          Show
          Jesse Yates added a comment - HBASE-5991 would be nice, but not necessary. Excited for it to go in though.
          Hide
          Ted Yu added a comment -

          Patch from Jesse.

          Show
          Ted Yu added a comment - Patch from Jesse.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12532749/hbase-5447-v8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2201//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532749/hbase-5447-v8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2201//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          @Jesse:
          HConstants.java has moved to hbase-common

          Can you rebase your patch ?

          Show
          Ted Yu added a comment - @Jesse: HConstants.java has moved to hbase-common Can you rebase your patch ?
          Hide
          Jesse Yates added a comment -

          Attaching rebased version.

          Show
          Jesse Yates added a comment - Attaching rebased version.
          Hide
          Lars Hofhansl added a comment -

          Rethinking the whole previous discussion... There is another thought:
          Simply never delete HFile, but instead always move them to an archive location instead. Then have an aynchronous thread using a pluggable policy to delete (or not) the HFiles from the archive location.

          That would completely sidestep any ZK synchronization issues. The removal of the archived files is not time critical and does not need to be synchronous on any path.
          Disadvantage is that a single thread in the master would have to do the cleanup (right?)

          Show
          Lars Hofhansl added a comment - Rethinking the whole previous discussion... There is another thought: Simply never delete HFile, but instead always move them to an archive location instead. Then have an aynchronous thread using a pluggable policy to delete (or not) the HFiles from the archive location. That would completely sidestep any ZK synchronization issues. The removal of the archived files is not time critical and does not need to be synchronous on any path. Disadvantage is that a single thread in the master would have to do the cleanup (right?)
          Hide
          Jesse Yates added a comment -

          @Lars: I was thinking that we should have this cleanup thread anyways (similar to the old logs). I could see moving the ZK stuff to just be monitored from that thread for what files it should delete or not (less a timeout on file movement). This would follow the same chaining that we are doing for the .oldlogs directory.

          I don't see it as all that bad to have an async thread from the master that does the cleanup (and we could remove the synchronous requirement in zk).

          I'm +1 on this - its not significantly more overhead than we have already and notably less overhead than the current implementation. Only downside is that the current implementation is pretty solid (been using it in my testing for snapshots and have yet to have a problem).

          Show
          Jesse Yates added a comment - @Lars: I was thinking that we should have this cleanup thread anyways (similar to the old logs). I could see moving the ZK stuff to just be monitored from that thread for what files it should delete or not (less a timeout on file movement). This would follow the same chaining that we are doing for the .oldlogs directory. I don't see it as all that bad to have an async thread from the master that does the cleanup (and we could remove the synchronous requirement in zk). I'm +1 on this - its not significantly more overhead than we have already and notably less overhead than the current implementation. Only downside is that the current implementation is pretty solid (been using it in my testing for snapshots and have yet to have a problem).
          Hide
          Jesse Yates added a comment -

          Anyone else have any objections to Lar's proposal? Otherwise I'd like to get hacking on it this afternoon.

          Show
          Jesse Yates added a comment - Anyone else have any objections to Lar's proposal? Otherwise I'd like to get hacking on it this afternoon.
          Hide
          Jesse Yates added a comment -

          Attaching revised version based on Lars' design. Ended up being way less code and complexity (+1).

          All the modified tests pass locally.

          Glad to throw this up on RB (esp given the size) if people want.

          Show
          Jesse Yates added a comment - Attaching revised version based on Lars' design. Ended up being way less code and complexity (+1). All the modified tests pass locally. Glad to throw this up on RB (esp given the size) if people want.
          Hide
          Jonathan Hsieh added a comment -

          RB please.

          Show
          Jonathan Hsieh added a comment - RB please.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12534010/hbase-5547-v9.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestColumnSeeking
          org.apache.hadoop.hbase.regionserver.TestScanner
          org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
          org.apache.hadoop.hbase.regionserver.TestKeepDeletes
          org.apache.hadoop.hbase.regionserver.TestMinVersions

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534010/hbase-5547-v9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestColumnSeeking org.apache.hadoop.hbase.regionserver.TestScanner org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface org.apache.hadoop.hbase.regionserver.TestKeepDeletes org.apache.hadoop.hbase.regionserver.TestMinVersions Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2295//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          Surprised that those tests are failing, looking into it (though they are failing locally too, so something is definitely up - looking into it.).

          Show
          Jesse Yates added a comment - Surprised that those tests are failing, looking into it (though they are failing locally too, so something is definitely up - looking into it.).
          Hide
          Jesse Yates added a comment -

          Attaching slightly updated v9 with fix for those broken tests - 3 line change did seem to warrant a new version.

          Show
          Jesse Yates added a comment - Attaching slightly updated v9 with fix for those broken tests - 3 line change did seem to warrant a new version.
          Hide
          Ted Yu added a comment -

          For BaseConfigurable:

          + * HBase version of Hadoop's Configured class that doesn't initialize the
          

          Why can't we use Hadoop's Configured class directly ?

          Looking forward to the patch on review board.

          Show
          Ted Yu added a comment - For BaseConfigurable: + * HBase version of Hadoop's Configured class that doesn't initialize the Why can't we use Hadoop's Configured class directly ? Looking forward to the patch on review board.
          Hide
          Jesse Yates added a comment -

          @Ted: Because in the constructor it does

          setConfiguration(null)
          

          which will screw up the delegate's configuration. This just seems simpler.

          Show
          Jesse Yates added a comment - @Ted: Because in the constructor it does setConfiguration( null ) which will screw up the delegate's configuration. This just seems simpler.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12534016/hbase-5547-v9.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534016/hbase-5547-v9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2297//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          https://reviews.apache.org/r/4633/ has been updated by Jesse.

          Show
          Ted Yu added a comment - https://reviews.apache.org/r/4633/ has been updated by Jesse.
          Hide
          Ted Yu added a comment -

          Currently hbase.master.cleaner.interval determines how often cleaning is done. Default is 60 seconds.
          In some situations, this may be short.

          Can we introduce capacity-based cleaning ?
          Operator can specify total hdfs space dedicated to the backup HFiles. If the threshold is exceeded, oldest backup HFiles would be cleaned up.

          For this JIRA, we can put cleanup policy in place, leaving implementation to a follow-up JIRA.

          Show
          Ted Yu added a comment - Currently hbase.master.cleaner.interval determines how often cleaning is done. Default is 60 seconds. In some situations, this may be short. Can we introduce capacity-based cleaning ? Operator can specify total hdfs space dedicated to the backup HFiles. If the threshold is exceeded, oldest backup HFiles would be cleaned up. For this JIRA, we can put cleanup policy in place, leaving implementation to a follow-up JIRA.
          Hide
          Jesse Yates added a comment -

          @Ted: you would still need to check the directory quite frequently or everytime you archive, you do some extra work adding the size of the files, both of which are unpalatable. You can get around this though by just cranking down the time between checks and then use a HFileArchiveSizeCleaner (or something like that). I can see that being useful - file an issue?

          Show
          Jesse Yates added a comment - @Ted: you would still need to check the directory quite frequently or everytime you archive, you do some extra work adding the size of the files, both of which are unpalatable. You can get around this though by just cranking down the time between checks and then use a HFileArchiveSizeCleaner (or something like that). I can see that being useful - file an issue?
          Hide
          Jesse Yates added a comment -

          Updated the diff on RB to the latest trunk. Any reviews/thoughts would be great!

          Show
          Jesse Yates added a comment - Updated the diff on RB to the latest trunk. Any reviews/thoughts would be great!
          Hide
          Lars Hofhansl added a comment -

          Will look at it this weekend!

          Show
          Lars Hofhansl added a comment - Will look at it this weekend!
          Hide
          stack added a comment -

          Added some review up on rb.

          Show
          stack added a comment - Added some review up on rb.
          Hide
          Lars Hofhansl added a comment -

          I think we should break this up into two parts:
          1. Don't delete the files, and the TTL HFile cleaner
          2. The special policy that has a user facing interface to retains HFiles for certain tables and relates that via ZK.

          #2 might even be in an example section, since I'd guess that most folks have special needs here anyway.

          As a suggestion, we might want to tack a timestamp into the rename HFile.

          I found it hard to wrap my head around this change, because of the refactoring of the log cleaner; but unifying the two is good.
          Great work!

          Show
          Lars Hofhansl added a comment - I think we should break this up into two parts: 1. Don't delete the files, and the TTL HFile cleaner 2. The special policy that has a user facing interface to retains HFiles for certain tables and relates that via ZK. #2 might even be in an example section, since I'd guess that most folks have special needs here anyway. As a suggestion, we might want to tack a timestamp into the rename HFile. I found it hard to wrap my head around this change, because of the refactoring of the log cleaner; but unifying the two is good. Great work!
          Hide
          Jesse Yates added a comment -

          1. Don't delete the files, and the TTL HFile cleaner
          2. The special policy that has a user facing interface to retains HFiles for certain tables and relates that via ZK.

          Clearly, we need to have #1 (and its in the current patch). I'd be fine to refactor the table-level retention via ZK into an example class/client (rather than as part of HBaseAdmin). However, splitting this into two tickets seems to be more work than its worth.

          @stack, @ted - what do you think?

          I found it hard to wrap my head around this change, because of the refactoring of the log cleaner; but unifying the two is good.

          Yeah, the hierarchy can make the reading a little odd, but it ends up being sweet that the HFileCleanerChore is just a constructor. Might argue for using factory, but that seemed a bit excessive for just two classes (maybe if we need to add another file cleaner chore later).

          Show
          Jesse Yates added a comment - 1. Don't delete the files, and the TTL HFile cleaner 2. The special policy that has a user facing interface to retains HFiles for certain tables and relates that via ZK. Clearly, we need to have #1 (and its in the current patch). I'd be fine to refactor the table-level retention via ZK into an example class/client (rather than as part of HBaseAdmin). However, splitting this into two tickets seems to be more work than its worth. @stack, @ted - what do you think? I found it hard to wrap my head around this change, because of the refactoring of the log cleaner; but unifying the two is good. Yeah, the hierarchy can make the reading a little odd, but it ends up being sweet that the HFileCleanerChore is just a constructor. Might argue for using factory, but that seemed a bit excessive for just two classes (maybe if we need to add another file cleaner chore later).
          Hide
          Lars Hofhansl added a comment -

          Should be in a 0.94 release.

          Show
          Lars Hofhansl added a comment - Should be in a 0.94 release.
          Hide
          Jesse Yates added a comment -

          My reservation to the 2nd part is that folks will have their own retention policy, which is going to be special to their organization and that HBase should not (try to) dictate that.
          Not feeling very strongly about this, though. As long as the TTL log cleaner is the default and the log term archiver is just a plugin (both of which are the case, right?).

          • Lars on RB

          My response:

          "both of which are the case, right?"
          yup.

          "On that note, I wonder if the Hlog cleaner and HFile cleaner should have a joint policy"
          That sounds subticket worthy. But interesting idea. It wouldn't be too hard to do with the existing setup, but out of scope

          If everyone is good with this, I'll push up a new version to RB that has the clean up from the reviews, abstracts out the client-zk stuff to an example.

          Show
          Jesse Yates added a comment - My reservation to the 2nd part is that folks will have their own retention policy, which is going to be special to their organization and that HBase should not (try to) dictate that. Not feeling very strongly about this, though. As long as the TTL log cleaner is the default and the log term archiver is just a plugin (both of which are the case, right?). Lars on RB My response: "both of which are the case, right?" yup. "On that note, I wonder if the Hlog cleaner and HFile cleaner should have a joint policy " That sounds subticket worthy. But interesting idea. It wouldn't be too hard to do with the existing setup, but out of scope If everyone is good with this, I'll push up a new version to RB that has the clean up from the reviews, abstracts out the client-zk stuff to an example.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536280/5547-v12.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 20 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536280/5547-v12.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2373//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          Attaching updated version based on trunk and addressing comments on RB. Also, had some replies to outstanding issues on RB.

          Show
          Jesse Yates added a comment - Attaching updated version based on trunk and addressing comments on RB. Also, had some replies to outstanding issues on RB.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536901/java_HBASE-5547_v13.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536901/java_HBASE-5547_v13.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2395//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Here is the link about InterruptedException handling:
          http://www.ibm.com/developerworks/java/library/j-jtp05236/index.html

          Take a look at Listing 3 under 'Don't swallow interrupts'

          Show
          Ted Yu added a comment - Here is the link about InterruptedException handling: http://www.ibm.com/developerworks/java/library/j-jtp05236/index.html Take a look at Listing 3 under 'Don't swallow interrupts'
          Hide
          Jesse Yates added a comment -

          Something Matteo mentioned in the latest review:

          MasterFileSystem contains deleteRegion() and deleteTable() that calls fs.delete() with the recursive flag on.
          This two methods get called by DeleteTableHandler (drop table).
          In a backup/snapshot situation we want to keep the regions/hfiles.

          Question is if we want to scope this in this patch or if we should do it in another patch, e.g. HBASE-6205?

          Show
          Jesse Yates added a comment - Something Matteo mentioned in the latest review: MasterFileSystem contains deleteRegion() and deleteTable() that calls fs.delete() with the recursive flag on. This two methods get called by DeleteTableHandler (drop table). In a backup/snapshot situation we want to keep the regions/hfiles. Question is if we want to scope this in this patch or if we should do it in another patch, e.g. HBASE-6205 ?
          Hide
          Jesse Yates added a comment -

          Question is if we want to scope this in this patch or if we should do it in another patch, e.g. HBASE-6205?

          Moving it in scope. It was a single line change and a ~100 lines of test. Posting new patch that also addresses Ted's recent comments.

          Show
          Jesse Yates added a comment - Question is if we want to scope this in this patch or if we should do it in another patch, e.g. HBASE-6205 ? Moving it in scope. It was a single line change and a ~100 lines of test. Posting new patch that also addresses Ted's recent comments.
          Hide
          Jesse Yates added a comment -

          Attaching latest from RB - addresses all Ted's comments and adding region archiving on table delete (+ testing!).

          Show
          Jesse Yates added a comment - Attaching latest from RB - addresses all Ted's comments and adding region archiving on table delete (+ testing!).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537114/java_HBASE-5547_v14.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 15 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.avro.TestAvroServer
          org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537114/java_HBASE-5547_v14.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 15 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.avro.TestAvroServer org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2407//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          From Jesse.

          Show
          Ted Yu added a comment - From Jesse.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537230/java_HBASE-5547_v15.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537230/java_HBASE-5547_v15.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2411//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Patch v16 corrects name of test class.

          Should be ready for integration.

          Show
          Ted Yu added a comment - Patch v16 corrects name of test class. Should be ready for integration.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537240/5547-v16.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 22 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537240/5547-v16.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 22 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2412//console This message is automatically generated.
          Hide
          Jesse Yates added a comment -

          Assuming changes between v15 and v16 are just what Ted mentioned on his last post on RB, then I'm good. Lets give it a day or so, before we integrate, so people have time to look at RB, if they haven't yet. Failed test doesn't apply to this code.

          Show
          Jesse Yates added a comment - Assuming changes between v15 and v16 are just what Ted mentioned on his last post on RB, then I'm good. Lets give it a day or so, before we integrate, so people have time to look at RB, if they haven't yet. Failed test doesn't apply to this code.
          Hide
          Ted Yu added a comment -

          I ran TestRegionServerCoprocessorExceptionWithAbort based on patch v16 and it passed.

          Show
          Ted Yu added a comment - I ran TestRegionServerCoprocessorExceptionWithAbort based on patch v16 and it passed.
          Hide
          Ted Yu added a comment -

          Will integrate in 3 hours if there is no objection.

          Show
          Ted Yu added a comment - Will integrate in 3 hours if there is no objection.
          Hide
          Lars Hofhansl added a comment -

          +1 on patch. Ted pinged me, that he is out already.
          Since this is a Salesforce patch, I should commit it anyway.

          Will do so as soon as I get to it.

          Jesse, do you have a feeling about how different a 0.94 patch would be?

          Show
          Lars Hofhansl added a comment - +1 on patch. Ted pinged me, that he is out already. Since this is a Salesforce patch, I should commit it anyway. Will do so as soon as I get to it. Jesse, do you have a feeling about how different a 0.94 patch would be?
          Hide
          Jesse Yates added a comment -

          @Lars I don't think it would be all that different. I'll take a crack next week (after dealing with the next round of HBASE-6055 stuff).

          Show
          Jesse Yates added a comment - @Lars I don't think it would be all that different. I'll take a crack next week (after dealing with the next round of HBASE-6055 stuff).
          Hide
          Lars Hofhansl added a comment -

          I also verified in a real setup, that an HFile is indeed archived and (by default) removed after 5 mins. Was thrown off first, because the table/region/cf directory is not removed when empty.
          Also made sure I can create/drop tables and then .META. is backed up correctly.

          So still +1

          Show
          Lars Hofhansl added a comment - I also verified in a real setup, that an HFile is indeed archived and (by default) removed after 5 mins. Was thrown off first, because the table/region/cf directory is not removed when empty. Also made sure I can create/drop tables and then .META. is backed up correctly. So still +1
          Hide
          Lars Hofhansl added a comment -

          Ok. On more question I asked just now on RB:

          From Matteo:

          MasterFileSystem contains deleteRegion() and deleteTable() that calls fs.delete() with the recursive flag on.
          This two methods get called by DeleteTableHandler (drop table).
          In a backup/snapshot situation we want to keep the regions/hfiles.

          My follow up question:

          I find that deleteRegion() was addressed, but not deleteTable().

          That means if a table is dropped the HFiles would be deleted and not archived.
          So it seems we should either:

          • also delete the table's archive directory (since it would be incomplete anyway).
          • archive all the HFile before deleting them.

          What do you think Jesse?

          Show
          Lars Hofhansl added a comment - Ok. On more question I asked just now on RB: From Matteo: MasterFileSystem contains deleteRegion() and deleteTable() that calls fs.delete() with the recursive flag on. This two methods get called by DeleteTableHandler (drop table). In a backup/snapshot situation we want to keep the regions/hfiles. My follow up question: I find that deleteRegion() was addressed, but not deleteTable(). That means if a table is dropped the HFiles would be deleted and not archived. So it seems we should either: also delete the table's archive directory (since it would be incomplete anyway). archive all the HFile before deleting them. What do you think Jesse?
          Hide
          Lars Hofhansl added a comment - - edited

          Or is it that all regions are first deleted anyway, and only then the deleteTable is called (in DeleteTableHandler.handleTableOperation)?

          Edit: Spelling

          Show
          Lars Hofhansl added a comment - - edited Or is it that all regions are first deleted anyway, and only then the deleteTable is called (in DeleteTableHandler.handleTableOperation)? Edit: Spelling
          Hide
          Ted Yu added a comment -

          I think the latest patch has addressed HFile archival when deleteRegion() is called.

          Backing up / restoring table can be addressed in HBASE-6055.

          Show
          Ted Yu added a comment - I think the latest patch has addressed HFile archival when deleteRegion() is called. Backing up / restoring table can be addressed in HBASE-6055 .
          Hide
          Lars Hofhansl added a comment -

          True, but if dropping a table just drops the latest HFiles to the floor and leaves a partial backup around this entire exercise is pointless.

          Anyway, from the code in DeleteTableHandler.handleTableOperation it looks like all regions are deleted first (using deleteRegion) and then the table directory is deleted, so it should be correct. Just making sure here.

          Show
          Lars Hofhansl added a comment - True, but if dropping a table just drops the latest HFiles to the floor and leaves a partial backup around this entire exercise is pointless. Anyway, from the code in DeleteTableHandler.handleTableOperation it looks like all regions are deleted first (using deleteRegion) and then the table directory is deleted, so it should be correct. Just making sure here.
          Hide
          Jesse Yates added a comment -

          I find that deleteRegion() was addressed, but not deleteTable().

          In the deletetablehandler, it first goes and deletes each region (which gets replaced with archiveRegion) and then once all the regions have been archived, we call deleteTable, which basically just "rm -rf"'s the entire directory.

          TL;DR - hfiles on table delete are archived, but not any more information about the state of the table. Wrote a unit test to check that too, so should be good.

          Show
          Jesse Yates added a comment - I find that deleteRegion() was addressed, but not deleteTable(). In the deletetablehandler, it first goes and deletes each region (which gets replaced with archiveRegion) and then once all the regions have been archived, we call deleteTable, which basically just "rm -rf"'s the entire directory. TL;DR - hfiles on table delete are archived, but not any more information about the state of the table. Wrote a unit test to check that too, so should be good.
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.96. Phfewww. Thanks for the patch and the persistence Jesse!
          Leaving open for the 0.94 part.

          Show
          Lars Hofhansl added a comment - Committed to 0.96. Phfewww. Thanks for the patch and the persistence Jesse! Leaving open for the 0.94 part.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3158 (See https://builds.apache.org/job/HBase-TRUNK/3158/)
          HBASE-5547 Don't delete HFiles when in "backup mode" (Jesse Yates) (Revision 1364203)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3158 (See https://builds.apache.org/job/HBase-TRUNK/3158/ ) HBASE-5547 Don't delete HFiles when in "backup mode" (Jesse Yates) (Revision 1364203) Result = FAILURE larsh : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Hide
          Lars Hofhansl added a comment -

          Hmm... From the failed run:

          Results :

          Failed tests: testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient): Archived HFiles should have gotten deleted, but didn't
          testRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation): expected:<0> but was:<5>

          Tests in error:
          testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation): java.io.FileNotFoundException: <https://builds.apache.org/job/HBase-TRUNK/ws/trunk/hbase-server/target/classes/hbase-default.xml> (Too many open files)

          The first one is suspicious. Could you have a look at it Jesse when you get a chance.

          Show
          Lars Hofhansl added a comment - Hmm... From the failed run: Results : Failed tests: testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient): Archived HFiles should have gotten deleted, but didn't testRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation): expected:<0> but was:<5> Tests in error: testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation): java.io.FileNotFoundException: < https://builds.apache.org/job/HBase-TRUNK/ws/trunk/hbase-server/target/classes/hbase-default.xml > (Too many open files) The first one is suspicious. Could you have a look at it Jesse when you get a chance.
          Hide
          Jesse Yates added a comment -

          Hmmm, looks like there was a NPE generated during a compaction - probably what broke the test, but its pretty early on, so maybe not. Looking into it to see what is going on...

          Show
          Jesse Yates added a comment - Hmmm, looks like there was a NPE generated during a compaction - probably what broke the test, but its pretty early on, so maybe not. Looking into it to see what is going on...
          Hide
          Jesse Yates added a comment -

          Attaching addendum that updates the test (removes the TTL cleaner to avoid timing issues) and fixes a couple of comments. Looks like a flapper given that build #3159 worked.

          Couldn't get the NPE to reproduce (didn't seem fatal though and was from a log statement).

          Ran the updated test 20x without failure.

          Show
          Jesse Yates added a comment - Attaching addendum that updates the test (removes the TTL cleaner to avoid timing issues) and fixes a couple of comments. Looks like a flapper given that build #3159 worked. Couldn't get the NPE to reproduce (didn't seem fatal though and was from a log statement). Ran the updated test 20x without failure.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537478/java_HBASE-5547.addendum
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

          -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537478/java_HBASE-5547.addendum against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2424//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #103 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/103/)
          HBASE-5547 Don't delete HFiles when in "backup mode" (Jesse Yates) (Revision 1364203)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
          • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #103 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/103/ ) HBASE-5547 Don't delete HFiles when in "backup mode" (Jesse Yates) (Revision 1364203) Result = FAILURE larsh : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java
          Hide
          Ted Yu added a comment -

          For the NPE, I think the null reference should have come from below code snippet:

                  (sf == null ? "none" : sf.getPath().getName()) +
                  ", size=" + (sf == null ? "none" :
                    StringUtils.humanReadableInt(sf.getReader().length()))
          

          Meaning either sf.getPath() or sf.getReader() was null.

          For the addendum, if TimeToLiveHFileCleaner is removed from TestZooKeeperTableArchiveClient, we should add test for it since it is a new class introduced by this feature.

          Show
          Ted Yu added a comment - For the NPE, I think the null reference should have come from below code snippet: (sf == null ? "none" : sf.getPath().getName()) + ", size=" + (sf == null ? "none" : StringUtils.humanReadableInt(sf.getReader().length())) Meaning either sf.getPath() or sf.getReader() was null. For the addendum, if TimeToLiveHFileCleaner is removed from TestZooKeeperTableArchiveClient, we should add test for it since it is a new class introduced by this feature.
          Hide
          Jesse Yates added a comment -

          Meaning either sf.getPath() or sf.getReader() was null.

          Looked through the code and can't say I see that was possible (though it seems the most likely given the setting/checking done on the store file). Looking through the test of the code, it doesn't look like an issue if the reader is null since it either gets closed immediately or is re-created when needed.

          The getPath() call could be bad, but its not doing anything funky and we would have seen a warning/error earlier in the run if that had actually been a problem.

          For the addendum, if TimeToLiveHFileCleaner is removed from TestZooKeeperTableArchiveClient, we should add test for it since it is a new class introduced by this feature.

          It is tested in o.a.h.h.master.cleaner.TestHFileCleaner. This change just cleans up the test so we don't have to worry about interaction with the TTLCleaner.

          Show
          Jesse Yates added a comment - Meaning either sf.getPath() or sf.getReader() was null. Looked through the code and can't say I see that was possible (though it seems the most likely given the setting/checking done on the store file). Looking through the test of the code, it doesn't look like an issue if the reader is null since it either gets closed immediately or is re-created when needed. The getPath() call could be bad, but its not doing anything funky and we would have seen a warning/error earlier in the run if that had actually been a problem. For the addendum, if TimeToLiveHFileCleaner is removed from TestZooKeeperTableArchiveClient, we should add test for it since it is a new class introduced by this feature. It is tested in o.a.h.h.master.cleaner.TestHFileCleaner. This change just cleans up the test so we don't have to worry about interaction with the TTLCleaner.
          Hide
          Jesse Yates added a comment -

          Hmm, looks like I broke the 2.0 build too. Here is the output from the logs there that seems pertinent:

          Regression

          org.apache.hadoop.hbase.avro.TestAvroServer.testTableAdminAndMetadata
          Failing for the past 1 build (Since Failed#103 )
          Took 32 ms.
          Error Message

          test timed out after 300000 milliseconds

          Stacktrace

          java.lang.Exception: test timed out after 300000 milliseconds
          at $Proxy26.scan(Unknown Source)
          at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:548)
          at org.apache.hadoop.hbase.avro.AvroServer$HBaseImpl.deleteTable(AvroServer.java:276)
          at org.apache.hadoop.hbase.avro.TestAvroServer.testTableAdminAndMetadata(TestAvroServer.java:122)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
          at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
          at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
          at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
          at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)

          Standard Output

          Formatting using clusterid: testClusterID

          Standard Error

          2012-07-22 13:02:19,346 WARN [pool-1-thread-1] conf.Configuration(737): hadoop.native.lib is deprecated. Instead, use io.native.lib.available
          2012-07-22 13:02:19,491 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(614): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s)
          2012-07-22 13:02:19,607 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(288): Created new mini-cluster data directory: /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce
          2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting test.cache.data to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/cache_data in system properties and HBase conf
          2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting hadoop.tmp.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/hadoop_tmp in system properties and HBase conf
          2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting hadoop.log.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/hadoop_logs in system properties and HBase conf
          2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapreduce.output.fileoutputformat.outputdir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_output in system properties and HBase conf
          2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.local.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_local in system properties and HBase conf
          2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.system.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_system in system properties and HBase conf
          2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.temp.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_temp in system properties and HBase conf
          2012-07-22 13:02:19,846 WARN [pool-1-thread-1] conf.Configuration(737): hadoop.configured.node.mapping is deprecated. Instead, use net.topology.configured.node.mapping
          2012-07-22 13:02:19,972 WARN [pool-1-thread-1] util.KerberosName(87): Kerberos krb5 configuration not found, setting default realm to empty
          2012-07-22 13:02:20,673 WARN [pool-1-thread-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
          2012-07-22 13:02:21,523 INFO [pool-1-thread-1] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
          2012-07-22 13:02:21,578 WARN [pool-1-thread-1] lib.StaticUserWebFilter(141): dfs.web.ugi should not be used. Instead, use hadoop.http.staticuser.user.
          2012-07-22 13:02:21,598 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26
          2012-07-22 13:02:21,625 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar:file:/home/hudson/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.0-alpha/hadoop-hdfs-2.0.0-alpha-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_39374_hdfs____.ksh5ik/webapp
          2012-07-22 13:02:21,777 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:39374
          2012-07-22 13:02:21,830 WARN [pool-1-thread-1] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          2012-07-22 13:02:21,848 WARN [pool-1-thread-1] lib.StaticUserWebFilter(141): dfs.web.ugi should not be used. Instead, use hadoop.http.staticuser.user.
          2012-07-22 13:02:21,851 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26
          2012-07-22 13:02:21,854 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar:file:/home/hudson/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.0-alpha/hadoop-hdfs-2.0.0-alpha-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_47954_datanode____.oum544/webapp
          2012-07-22 13:02:21,933 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:47954
          2012-07-22 13:02:22,755 WARN [pool-1-thread-1] conf.Configuration(737): fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-07-22 13:02:22,827 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(197): Started MiniZK Cluster and connect 1 ZK server on client port: 59288
          2012-07-22 13:02:22,978 DEBUG [pool-1-thread-1] util.FSUtils(438): Created version file at hdfs://localhost:52085/user/hudson/hbase with version=7
          2012-07-22 13:02:23,026 DEBUG [pool-1-thread-1] client.HConnectionManager(2483): Set serverside HConnection retries=100
          2012-07-22 13:02:23,050 INFO [pool-1-thread-1] ipc.HBaseRpcMetrics(66): Initializing RPC Metrics with hostName=HMaster, port=55921
          2012-07-22 13:02:23,095 WARN [pool-1-thread-1] conf.Configuration(737): mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
          2012-07-22 13:02:23,102 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(103): master:55921 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:23,122 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:23,238 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:23,242 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(339): master:55921-0x138aec80e530000 connected
          2012-07-22 13:02:23,322 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: revision
          2012-07-22 13:02:23,322 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsUser
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsDate
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsUrl
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: date
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsRevision
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: user
          2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsVersion
          2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: url
          2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: version
          2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(161): new MBeanInfo
          2012-07-22 13:02:23,324 ERROR [pool-1-thread-1] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:23,325 ERROR [pool-1-thread-1] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:23,325 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(161): new MBeanInfo
          2012-07-22 13:02:23,325 INFO [pool-1-thread-1] metrics.MasterMetrics(102): Initialized
          2012-07-22 13:02:23,356 DEBUG [pool-1-thread-1] client.HConnectionManager(2483): Set serverside HConnection retries=100
          2012-07-22 13:02:23,359 INFO [pool-1-thread-1] ipc.HBaseRpcMetrics(66): Initializing RPC Metrics with hostName=MiniHBaseCluster$MiniHBaseClusterRegionServer, port=56523
          2012-07-22 13:02:23,401 INFO [pool-1-thread-1] hfile.CacheConfig(350): Allocating LruBlockCache with maximum size 422.2m
          2012-07-22 13:02:23,411 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/shutdown does not exist. Watcher is set.
          2012-07-22 13:02:23,421 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ActiveMasterManager(150): Deleting ZNode for /hbase/backup-masters/vesta.apache.org,55921,1342962143091 from backup master directory
          2012-07-22 13:02:23,431 WARN [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(142): Node /hbase/backup-masters/vesta.apache.org,55921,1342962143091 already deleted, and this is not a retry
          2012-07-22 13:02:23,432 WARN [Master:0;vesta.apache.org,55921,1342962143091] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set).
          2012-07-22 13:02:23,432 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ActiveMasterManager(159): Master=vesta.apache.org,55921,1342962143091
          2012-07-22 13:02:23,434 WARN [Master:0;vesta.apache.org,55921,1342962143091] conf.Configuration(737): fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-07-22 13:02:23,438 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(175): timeout = 25000
          2012-07-22 13:02:23,438 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(176): unassigned timeout = 180000
          2012-07-22 13:02:23,441 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(806): found 0 orphan tasks and 0 rescan nodes
          2012-07-22 13:02:23,504 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(567): Created cluster ID file at hdfs://localhost:52085/user/hudson/hbase/hbase.id with ID: f285209f-4f02-4b1d-817f-d5611467793a
          2012-07-22 13:02:23,508 INFO [pool-1-thread-1] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RegionServer:0;vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:23,512 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(103): regionserver:56523 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:23,513 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:23,521 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:23,522 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(339): regionserver:56523-0x138aec80e530001 connected
          2012-07-22 13:02:23,522 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(236): regionserver:56523-0x138aec80e530001 Set watcher on existing znode /hbase/master
          2012-07-22 13:02:23,527 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 35 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A\x1D\x0A\x10vesta.apa...
          2012-07-22 13:02:23,527 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/shutdown does not exist. Watcher is set.
          2012-07-22 13:02:23,539 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.MasterFileSystem(394): BOOTSTRAP: creating ROOT and first META regions
          2012-07-22 13:02:23,539 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(3781): creating HRegion ROOT HTD == 'ROOT',

          Unknown macro: {METHOD => 'table_att', IS_META => 'true', IS_ROOT => 'true'}

          ,

          Unknown macro: {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'false'}

          RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == ROOT
          2012-07-22 13:02:23,580 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
          2012-07-22 13:02:23,622 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840
          2012-07-22 13:02:23,622 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.logs/hlog.1342962143580, compression=false
          2012-07-22 13:02:23,622 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(652): for /user/hudson/hbase/ROOT/70236052/.logs/hlog.1342962143580
          2012-07-22 13:02:23,623 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(500): Using getNumCurrentReplicas--HDFS-826
          2012-07-22 13:02:23,639 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(464): Instantiated ROOT,,0.70236052
          2012-07-22 13:02:23,643 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/.regioninfo with permission=rwxrwxrwx
          2012-07-22 13:02:23,730 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(224): hbase.hstore.compaction.min = 3
          2012-07-22 13:02:23,738 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] util.ChecksumType$2(68): Checksum using org.apache.hadoop.util.PureJavaCrc32
          2012-07-22 13:02:23,738 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] util.ChecksumType$3(111): Checksum can use org.apache.hadoop.util.PureJavaCrc32C
          2012-07-22 13:02:23,748 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(609): Onlined ROOT,,0.70236052; next sequenceid=1
          2012-07-22 13:02:23,748 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(3781): creating HRegion .META. HTD == '.META.',

          Unknown macro: {METHOD => 'table_att', IS_META => 'true'}

          ,

          RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == .META.
          2012-07-22 13:02:23,787 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
          2012-07-22 13:02:23,803 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840
          2012-07-22 13:02:23,803 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.logs/hlog.1342962143787, compression=false
          2012-07-22 13:02:23,803 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(652): for /user/hudson/hbase/.META./1028785192/.logs/hlog.1342962143787
          2012-07-22 13:02:23,803 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(500): Using getNumCurrentReplicas--HDFS-826
          2012-07-22 13:02:23,804 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(464): Instantiated .META.,,1.1028785192
          2012-07-22 13:02:23,806 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.tmp/.regioninfo with permission=rwxrwxrwx
          2012-07-22 13:02:23,882 INFO [StoreOpenerThread-.META.,,1.1028785192-1] regionserver.Store(224): hbase.hstore.compaction.min = 3
          2012-07-22 13:02:23,887 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(609): Onlined .META.,,1.1028785192; next sequenceid=1
          2012-07-22 13:02:23,909 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(941): Closing ROOT,,0.70236052: disabling compactions & flushes
          2012-07-22 13:02:23,909 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(962): Updates disabled for region ROOT,,0.70236052
          2012-07-22 13:02:23,910 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1468): Started memstore flush for ROOT,,0.70236052, current region memstore size 360
          2012-07-22 13:02:23,910 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1516): Finished snapshotting ROOT,,0.70236052, commencing wait for mvcc, flushsize=360
          2012-07-22 13:02:23,911 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1526): Finished snapshotting, commencing flushing stores
          2012-07-22 13:02:23,932 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0 with permission=rwxrwxrwx
          2012-07-22 13:02:23,953 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] hfile.HFileWriterV2(143): Initialized with CacheConfig:enabled [cacheDataOnRead=false] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
          2012-07-22 13:02:23,956 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.StoreFile$Writer(1001): Delete Family Bloom filter type for hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0: CompoundBloomFilterWriter
          2012-07-22 13:02:23,995 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.StoreFile$Writer(1221): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0)
          2012-07-22 13:02:23,996 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(785): Flushed , sequenceid=2, memsize=360, into tmp file hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0
          2012-07-22 13:02:24,013 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(810): Renaming flushed file at hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0 to hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0
          2012-07-22 13:02:24,031 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(833): Added hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0, entries=2, sequenceid=2, filesize=848
          2012-07-22 13:02:24,032 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1600): Finished memstore flush of ~360/360, currentsize=0/0 for region ROOT,,0.70236052 in 122ms, sequenceid=2, compaction requested=false
          2012-07-22 13:02:24,035 INFO [StoreCloserThread--ROOT-,,0.70236052-1] regionserver.Store(670): Closed info
          2012-07-22 13:02:24,035 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1010): Closed ROOT,,0.70236052
          2012-07-22 13:02:24,035 INFO [Master:0;vesta.apache.org,55921,1342962143091.logSyncer] wal.HLog$LogSyncer(1237): Master:0;vesta.apache.org,55921,1342962143091.logSyncer exiting
          2012-07-22 13:02:24,036 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/.logs
          2012-07-22 13:02:24,061 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/ROOT/70236052/.oldlogs
          2012-07-22 13:02:24,073 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(941): Closing .META.,,1.1028785192: disabling compactions & flushes
          2012-07-22 13:02:24,074 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(962): Updates disabled for region .META.,,1.1028785192
          2012-07-22 13:02:24,074 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.Store(670): Closed info
          2012-07-22 13:02:24,074 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1010): Closed .META.,,1.1028785192
          2012-07-22 13:02:24,074 INFO [Master:0;vesta.apache.org,55921,1342962143091.logSyncer] wal.HLog$LogSyncer(1237): Master:0;vesta.apache.org,55921,1342962143091.logSyncer exiting
          2012-07-22 13:02:24,075 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.logs
          2012-07-22 13:02:24,109 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/.META./1028785192/.oldlogs
          2012-07-22 13:02:24,619 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(103): hconnection 0xcf7fd0 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:24,620 INFO [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:24,629 DEBUG [Master:0;vesta.apache.org,55921,1342962143091-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0xcf7fd0 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:24,630 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8...
          2012-07-22 13:02:24,630 INFO [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a
          2012-07-22 13:02:24,630 DEBUG [Master:0;vesta.apache.org,55921,1342962143091-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0xcf7fd0-0x138aec80e530002 connected
          2012-07-22 13:02:24,632 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] catalog.CatalogTracker(240): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@cf5006
          2012-07-22 13:02:24,633 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/root-region-server does not exist. Watcher is set.
          2012-07-22 13:02:24,633 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/unassigned/1028785192 does not exist. Watcher is set.
          2012-07-22 13:02:24,654 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/shutdown
          2012-07-22 13:02:24,654 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/shutdown
          2012-07-22 13:02:24,655 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(524): Server active/primary master; vesta.apache.org,55921,1342962143091, sessionid=0x138aec80e530000, cluster-up flag was=false
          2012-07-22 13:02:24,655 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 34 byte(s) of data from znode /hbase/shutdown and set watcher; PBUF\x0A\x1CSun Jul 22 13:02:...
          2012-07-22 13:02:24,655 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 34 byte(s) of data from znode /hbase/shutdown and set watcher; PBUF\x0A\x1CSun Jul 22 13:02:...
          2012-07-22 13:02:24,659 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(103): hconnection 0xe8f27e opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:24,660 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5
          2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5
          2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_SERVER_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=3, maxPoolSize=3
          2012-07-22 13:02:24,663 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_META_SERVER_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5
          2012-07-22 13:02:24,663 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=1, maxPoolSize=1
          2012-07-22 13:02:24,667 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(1034): Started service threads
          2012-07-22 13:02:24,667 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(627): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms.
          2012-07-22 13:02:24,669 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0xe8f27e Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(1142): hconnection 0xe8f27e Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8...
          2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0xe8f27e-0x138aec80e530003 connected
          2012-07-22 13:02:24,670 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a
          2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] catalog.CatalogTracker(240): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5b701a
          2012-07-22 13:02:24,671 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/root-region-server does not exist. Watcher is set.
          2012-07-22 13:02:24,672 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/unassigned/1028785192 does not exist. Watcher is set.
          2012-07-22 13:02:24,674 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.MemStoreFlusher(112): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g
          2012-07-22 13:02:24,676 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer$CompactionChecker(1216): Runs every 16mins, 40sec
          2012-07-22 13:02:24,709 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1828): Attempting connect to Master server at vesta.apache.org,55921,1342962143091
          2012-07-22 13:02:24,755 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1837): Connected to master at vesta.apache.org/67.195.138.9:55921
          2012-07-22 13:02:24,755 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1883): Telling master at vesta.apache.org,55921,1342962143091 that we are up with port=56523, startcode=1342962143399
          2012-07-22 13:02:24,759 INFO [IPC Server handler 2 on 55921] master.ServerManager(270): Registering server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:24,764 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1063): Config from master: hbase.rootdir=hdfs://localhost:52085/user/hudson/hbase
          2012-07-22 13:02:24,764 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1063): Config from master: fs.default.name=hdfs://localhost:52085
          2012-07-22 13:02:24,764 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1056): Master passed us hostname to use. Was=vesta.apache.org, Now=vesta.apache.org
          2012-07-22 13:02:24,767 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(644): Finished waiting for region servers count to settle; checked in 1, slept for 100 ms, expecting minimum of 1, maximum of 1, master is running.
          2012-07-22 13:02:24,768 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs
          2012-07-22 13:02:24,768 WARN [RegionServer:0;vesta.apache.org,56523,1342962143399] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set).
          2012-07-22 13:02:24,769 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(236): master:55921-0x138aec80e530000 Set watcher on existing znode /hbase/rs/vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:24,772 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1277): logdir=hdfs://localhost:52085/user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:24,786 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
          2012-07-22 13:02:24,804 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840
          2012-07-22 13:02:24,804 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399/vesta.apache.org%2C56523%2C1342962143399.1342962144786, compression=false
          2012-07-22 13:02:24,804 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(652): for /user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399/vesta.apache.org%2C56523%2C1342962143399.1342962144786
          2012-07-22 13:02:24,804 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(500): Using getNumCurrentReplicas--HDFS-826
          2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,808 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,808 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
          2012-07-22 13:02:24,808 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.MetricsMBeanBase(161): new MBeanInfo
          2012-07-22 13:02:24,808 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.RegionServerMetrics(326): Initialized
          2012-07-22 13:02:24,809 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.MetricsMBeanBase(161): new MBeanInfo
          2012-07-22 13:02:24,810 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.RegionServerDynamicMetrics(91): java.lang.NoSuchMethodException: org.apache.hadoop.hbase.metrics.MetricsMBeanBase.updateMbeanInfoIfMetricsListChanged()
          2012-07-22 13:02:24,810 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_REGION-vesta.apache.org,56523,1342962143399, corePoolSize=3, maxPoolSize=3
          2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_ROOT-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1
          2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_META-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1
          2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399, corePoolSize=3, maxPoolSize=3
          2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_ROOT-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1
          2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_META-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1
          2012-07-22 13:02:24,819 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1095): Serving as vesta.apache.org,56523,1342962143399, RPC listening on vesta.apache.org/67.195.138.9:56523, sessionid=0x138aec80e530001
          2012-07-22 13:02:24,819 INFO [SplitLogWorker-vesta.apache.org,56523,1342962143399] regionserver.SplitLogWorker(135): SplitLogWorker vesta.apache.org,56523,1342962143399 starting
          2012-07-22 13:02:24,820 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(2353): Registered RegionServer MXBean
          2012-07-22 13:02:25,778 INFO [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RootRegionTracker(157): Unsetting ROOT region location in ZooKeeper
          2012-07-22 13:02:25,783 WARN [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(142): Node /hbase/root-region-server already deleted, and this is not a retry
          2012-07-22 13:02:25,785 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKAssign(272): master:55921-0x138aec80e530000 Creating (or updating) unassigned node for 70236052 with OFFLINE state
          2012-07-22 13:02:25,799 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1732): Setting table ROOT to ENABLED state.
          2012-07-22 13:02:25,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1976): No previous transition plan was found (or we are ignoring an existing plan) for ROOT,,0.70236052 so generated a random one; hri=ROOT,,0.70236052, src=, dest=vesta.apache.org,56523,1342962143399; 1 (online=1, available=1) available servers
          2012-07-22 13:02:25,819 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1751): Assigning region ROOT,,0.70236052 to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,846 INFO [PRI IPC Server handler 2 on 56523] regionserver.HRegionServer(3428): Received request to open region: ROOT,,0.70236052 on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,848 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ROOT from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:25,849 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 57 byte(s) of data from znode /hbase/unassigned/70236052; data=region=ROOT,,0, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962145788, payload.length=0
          2012-07-22 13:02:25,854 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052
          2012-07-22 13:02:25,854 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:25,855 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(3851): Opening region:

          Unknown macro: {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}

          2012-07-22 13:02:25,855 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region=ROOT,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145849, payload.length=0
          2012-07-22 13:02:25,856 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=70236052/ROOT, current state from RIT=ROOT,,0.70236052 state=PENDING_OPEN, ts=1342962145819, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,860 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(464): Instantiated ROOT,,0.70236052
          2012-07-22 13:02:25,900 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(224): hbase.hstore.compaction.min = 3
          2012-07-22 13:02:25,912 DEBUG [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(428): loaded hdfs://localhost:52085/user/hudson/hbase/ROOT/70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0, isReference=false, isBulkLoadResult=false, seqid=2, majorCompaction=false
          2012-07-22 13:02:25,915 INFO [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(609): Onlined ROOT,,0.70236052; next sequenceid=3
          2012-07-22 13:02:25,916 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ROOT from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:25,916 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=region=ROOT,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145849, payload.length=0
          2012-07-22 13:02:25,922 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052
          2012-07-22 13:02:25,922 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:25,923 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region=ROOT,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145917, payload.length=0
          2012-07-22 13:02:25,923 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=70236052/ROOT, current state from RIT=ROOT,,0.70236052 state=OPENING, ts=1342962145849, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,923 INFO [PostOpenDeployTasks:70236052] regionserver.HRegionServer(1644): Post open deploy tasks for region=ROOT,,0.70236052, daughter=false
          2012-07-22 13:02:25,923 INFO [PostOpenDeployTasks:70236052] zookeeper.RootRegionTracker(123): Setting ROOT region location in ZooKeeper as vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,932 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/root-region-server
          2012-07-22 13:02:25,932 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/root-region-server
          2012-07-22 13:02:25,933 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,933 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,933 INFO [PostOpenDeployTasks:70236052] regionserver.HRegionServer(1669): Done with post open deploy task for region=ROOT,,0.70236052, daughter=false
          2012-07-22 13:02:25,934 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ROOT from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:25,934 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=region=ROOT,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145917, payload.length=0
          2012-07-22 13:02:25,944 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052
          2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(286): region transitioned to opened in zookeeper:

          , server: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(142): Opened ROOT,,0.70236052 on server:vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,945 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region=ROOT,,0, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145935, payload.length=0
          2012-07-22 13:02:25,945 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENED, server=vesta.apache.org,56523,1342962143399, region=70236052/ROOT, current state from RIT=ROOT,,0.70236052 state=OPENING, ts=1342962145917, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,946 INFO [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] handler.OpenedRegionHandler(147): Handling OPENED event for ROOT,,0.70236052 from vesta.apache.org,56523,1342962143399; deleting unassigned node
          2012-07-22 13:02:25,946 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for 70236052 that is in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:25,947 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=region=ROOT,,0, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145935, payload.length=0
          2012-07-22 13:02:25,956 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/70236052
          2012-07-22 13:02:25,956 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKAssign(521): master:55921-0x138aec80e530000 Successfully deleted unassigned node for region 70236052 in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:25,956 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(1191): The znode of region ROOT,,0.70236052 has been deleted.
          2012-07-22 13:02:25,957 INFO [pool-1-thread-1-EventThread] master.AssignmentManager(1204): The master has opened the region ROOT,,0.70236052 that was online on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,957 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,958 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(800): ROOT assigned=1, rit=false, location=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,964 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:25,965 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,965 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,967 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:25,967 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:25,967 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,067 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,068 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,068 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,068 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,069 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,069 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,121 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,122 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,122 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,122 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,123 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,123 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,175 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,175 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,175 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,176 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,176 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,176 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,228 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,229 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,229 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,229 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,230 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,230 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,282 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,282 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,282 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,283 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,283 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,283 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,337 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,337 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,337 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,338 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,338 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,338 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,390 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,391 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,391 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,391 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,392 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,392 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,443 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,444 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,444 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,444 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,445 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,445 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,497 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,498 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,498 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,498 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,498 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,499 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,551 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,552 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,552 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,552 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,553 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,553 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,605 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,605 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,605 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,606 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,606 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,606 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,658 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,659 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,659 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,659 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,660 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,660 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,711 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,712 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,712 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,712 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,713 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,713 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,765 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,765 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,766 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,766 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,766 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,766 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,818 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,820 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,820 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,872 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,873 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,873 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,873 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,874 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,874 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,925 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,926 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,926 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,926 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,927 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,927 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,979 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,979 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,979 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,980 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,980 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,980 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,982 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,982 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,983 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,983 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:26,983 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,984 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:26,985 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKAssign(272): master:55921-0x138aec80e530000 Creating (or updating) unassigned node for 1028785192 with OFFLINE state
          2012-07-22 13:02:27,020 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,020 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,021 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962146985, payload.length=0
          2012-07-22 13:02:27,021 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962146985, payload.length=0
          2012-07-22 13:02:27,021 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1732): Setting table .META. to ENABLED state.
          2012-07-22 13:02:27,021 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=M_ZK_REGION_OFFLINE, server=vesta.apache.org,55921,1342962143091, region=1028785192/.META., current state from RIT=.META.,,1.1028785192 state=OFFLINE, ts=1342962146985, server=null
          2012-07-22 13:02:27,022 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962146985, payload.length=0
          2012-07-22 13:02:27,037 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1976): No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=vesta.apache.org,56523,1342962143399; 1 (online=1, available=1) available servers
          2012-07-22 13:02:27,037 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1751): Assigning region .META.,,1.1028785192 to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,037 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,037 INFO [PRI IPC Server handler 5 on 56523] regionserver.HRegionServer(3428): Received request to open region: .META.,,1.1028785192 on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,038 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 1028785192/.META. from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:27,039 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,039 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data=region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962146985, payload.length=0
          2012-07-22 13:02:27,040 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,040 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,040 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,056 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,056 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,056 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 1028785192 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:27,057 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(3851): Opening region:

          Unknown macro: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

          2012-07-22 13:02:27,057 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,057 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147040, payload.length=0
          2012-07-22 13:02:27,057 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,057 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147040, payload.length=0
          2012-07-22 13:02:27,057 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=1028785192/.META., current state from RIT=.META.,,1.1028785192 state=PENDING_OPEN, ts=1342962147037, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,057 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(464): Instantiated .META.,,1.1028785192
          2012-07-22 13:02:27,058 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147040, payload.length=0
          2012-07-22 13:02:27,062 INFO [StoreOpenerThread-.META.,,1.1028785192-1] regionserver.Store(224): hbase.hstore.compaction.min = 3
          2012-07-22 13:02:27,066 INFO [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(609): Onlined .META.,,1.1028785192; next sequenceid=1
          2012-07-22 13:02:27,066 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:27,067 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147040, payload.length=0
          2012-07-22 13:02:27,076 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,076 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,076 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:27,077 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1644): Post open deploy tasks for region=.META.,,1.1028785192, daughter=false
          2012-07-22 13:02:27,077 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147067, payload.length=0
          2012-07-22 13:02:27,077 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147067, payload.length=0
          2012-07-22 13:02:27,077 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=1028785192/.META., current state from RIT=.META.,,1.1028785192 state=OPENING, ts=1342962147040, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,078 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147067, payload.length=0
          2012-07-22 13:02:27,078 DEBUG [PostOpenDeployTasks:1028785192] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xe8f27e
          2012-07-22 13:02:27,079 DEBUG [PostOpenDeployTasks:1028785192] zookeeper.ZKUtil(1142): hconnection 0xe8f27e-0x138aec80e530003 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,079 DEBUG [PostOpenDeployTasks:1028785192] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xe8f27e; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,080 DEBUG [PostOpenDeployTasks:1028785192] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xe8f27e
          2012-07-22 13:02:27,080 DEBUG [PostOpenDeployTasks:1028785192] zookeeper.ZKUtil(1142): hconnection 0xe8f27e-0x138aec80e530003 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,080 DEBUG [PostOpenDeployTasks:1028785192] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xe8f27e; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,279 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,281 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,281 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,281 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,282 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,282 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,286 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,286 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,286 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,287 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,287 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,287 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,302 INFO [PostOpenDeployTasks:1028785192] catalog.MetaEditor(263): Updated row .META.,,1.1028785192 with server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,302 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1669): Done with post open deploy task for region=.META.,,1.1028785192, daughter=false
          2012-07-22 13:02:27,303 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:27,303 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147067, payload.length=0
          2012-07-22 13:02:27,307 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,307 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,307 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:27,308 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(286): region transitioned to opened in zookeeper:

          , server: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,308 DEBUG [RS_OPEN_META-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(142): Opened .META.,,1.1028785192 on server:vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,308 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147303, payload.length=0
          2012-07-22 13:02:27,308 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147303, payload.length=0
          2012-07-22 13:02:27,308 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENED, server=vesta.apache.org,56523,1342962143399, region=1028785192/.META., current state from RIT=.META.,,1.1028785192 state=OPENING, ts=1342962147067, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,309 INFO [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-1] handler.OpenedRegionHandler(147): Handling OPENED event for .META.,,1.1028785192 from vesta.apache.org,56523,1342962143399; deleting unassigned node
          2012-07-22 13:02:27,309 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-1] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:27,309 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147303, payload.length=0
          2012-07-22 13:02:27,310 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-1] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962147303, payload.length=0
          2012-07-22 13:02:27,319 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,319 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192
          2012-07-22 13:02:27,319 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-1] zookeeper.ZKAssign(521): master:55921-0x138aec80e530000 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:27,319 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(1191): The znode of region .META.,,1.1028785192 has been deleted.
          2012-07-22 13:02:27,319 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] catalog.CatalogTracker(528): Current cached META location, null, is not valid, resetting
          2012-07-22 13:02:27,320 INFO [pool-1-thread-1-EventThread] master.AssignmentManager(1204): The master has opened the region .META.,,1.1028785192 that was online on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,320 DEBUG [pool-1-thread-1-EventThread] catalog.CatalogTracker(528): Current cached META location, null, is not valid, resetting
          2012-07-22 13:02:27,320 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,321 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,321 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,321 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,321 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,322 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,328 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] catalog.CatalogTracker(540): Set new cached META location: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,328 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(828): .META. assigned=2, rit=false, location=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,329 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,330 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,330 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,331 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(94): Creating scanner over ROOT starting at key ''
          2012-07-22 13:02:27,331 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,332 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,333 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,333 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,342 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(192): Finished with scanning at

          Unknown macro: {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}

          2012-07-22 13:02:27,342 INFO [Master:0;vesta.apache.org,55921,1342962143091] catalog.MetaMigrationRemovingHTD(225): Meta version=0; migrated=true
          2012-07-22 13:02:27,342 INFO [Master:0;vesta.apache.org,55921,1342962143091] catalog.MetaMigrationRemovingHTD(246): ROOT/Meta already up-to date with new HRI.
          2012-07-22 13:02:27,343 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xcf7fd0
          2012-07-22 13:02:27,344 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0-0x138aec80e530002 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,344 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xcf7fd0; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,349 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(1335): Cached location for .META.,,1.1028785192 is vesta.apache.org:56523
          2012-07-22 13:02:27,349 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(94): Creating scanner over .META. starting at key ''
          2012-07-22 13:02:27,349 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,352 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(192): Finished with scanning at

          Unknown macro: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

          2012-07-22 13:02:27,353 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(467): Clean cluster startup. Assigning userregions
          2012-07-22 13:02:27,353 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKAssign(539): master:55921-0x138aec80e530000 Deleting any existing unassigned nodes
          2012-07-22 13:02:27,354 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(94): Creating scanner over .META. starting at key ''
          2012-07-22 13:02:27,354 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,357 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(192): Finished with scanning at

          2012-07-22 13:02:27,358 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(94): Creating scanner over .META. starting at key ''
          2012-07-22 13:02:27,358 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,361 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] client.ClientScanner(192): Finished with scanning at

          Unknown macro: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

          2012-07-22 13:02:27,365 DEBUG [vesta.apache.org,55921,1342962143091-CatalogJanitor] client.ClientScanner(94): Creating scanner over .META. starting at key ''
          2012-07-22 13:02:27,366 DEBUG [vesta.apache.org,55921,1342962143091-CatalogJanitor] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,367 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(2262): Registered HMaster MXBean
          2012-07-22 13:02:27,368 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(693): Master has completed initialization
          2012-07-22 13:02:27,370 DEBUG [vesta.apache.org,55921,1342962143091-CatalogJanitor] client.ClientScanner(192): Finished with scanning at

          2012-07-22 13:02:27,370 DEBUG [vesta.apache.org,55921,1342962143091-CatalogJanitor] master.CatalogJanitor(170): Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
          2012-07-22 13:02:27,381 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(103): hconnection 0xf49ac3 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:27,382 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:27,387 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0xf49ac3 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:27,388 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1142): hconnection 0xf49ac3 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8...
          2012-07-22 13:02:27,388 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a
          2012-07-22 13:02:27,389 DEBUG [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xf49ac3
          2012-07-22 13:02:27,389 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0xf49ac3-0x138aec80e530004 connected
          2012-07-22 13:02:27,389 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1142): hconnection 0xf49ac3-0x138aec80e530004 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,389 DEBUG [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xf49ac3; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:27,394 DEBUG [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1335): Cached location for .META.,,1.1028785192 is vesta.apache.org:56523
          2012-07-22 13:02:27,394 DEBUG [pool-1-thread-1] client.ClientScanner(94): Creating scanner over .META. starting at key ''
          2012-07-22 13:02:27,394 DEBUG [pool-1-thread-1] client.ClientScanner(205): Advancing internal scanner to startKey at ''
          2012-07-22 13:02:27,397 DEBUG [pool-1-thread-1] client.ClientScanner(192): Finished with scanning at

          Unknown macro: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

          2012-07-22 13:02:27,397 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1544): Closing zookeeper sessionid=0x138aec80e530004
          2012-07-22 13:02:27,415 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(103): hconnection 0x1fd9b49 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:27,416 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:27,426 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0x1fd9b49 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:27,427 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8...
          2012-07-22 13:02:27,427 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a
          2012-07-22 13:02:27,427 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(673): Minicluster is up
          2012-07-22 13:02:27,428 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0x1fd9b49-0x138aec80e530005 connected
          2012-07-22 13:02:27,437 INFO [pool-1-thread-1] hbase.ResourceChecker(145): before avro.TestAvroServer#testTableAdminAndMetadata: 138 threads, 319 file descriptors 3 connections,
          2012-07-22 13:02:27,550 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 35 byte(s) of data from znode /hbase/master; data=PBUF\x0A\x1D\x0A\x10vesta.apa...
          2012-07-22 13:02:27,908 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 35 byte(s) of data from znode /hbase/master; data=PBUF\x0A\x1D\x0A\x10vesta.apa...
          2012-07-22 13:02:27,941 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableA,,'
          2012-07-22 13:02:27,941 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableA,,'
          2012-07-22 13:02:27,944 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(192): Finished with scanning at

          2012-07-22 13:02:28,116 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.CreateTableHandler(128): Attempting to create the table tableA
          2012-07-22 13:02:28,117 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] util.FSUtils(1114): hdfs://localhost:52085/user/hudson/hbase/tableA doesn't exist
          2012-07-22 13:02:28,120 DEBUG [Thread-137] client.MetaScanner(202): Scanning .META. starting at row=tableA,,00000000000000 for max=2147483647 rows using hconnection 0x1fd9b49
          2012-07-22 13:02:28,120 DEBUG [Thread-137] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0x1fd9b49
          2012-07-22 13:02:28,120 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,121 DEBUG [Thread-137] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0x1fd9b49; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,125 DEBUG [Thread-137] client.HConnectionManager$HConnectionImplementation(1335): Cached location for .META.,,1.1028785192 is vesta.apache.org:56523
          2012-07-22 13:02:28,169 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(3781): creating HRegion tableA HTD == 'tableA' RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == tableA
          2012-07-22 13:02:28,217 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
          2012-07-22 13:02:28,242 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840
          2012-07-22 13:02:28,242 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/tableA/a697a37d317a95f9881b48cc409eafb1/.logs/hlog.1342962148217, compression=false
          2012-07-22 13:02:28,242 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(652): for /user/hudson/hbase/tableA/a697a37d317a95f9881b48cc409eafb1/.logs/hlog.1342962148217
          2012-07-22 13:02:28,242 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(500): Using getNumCurrentReplicas--HDFS-826
          2012-07-22 13:02:28,242 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(464): Instantiated tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.
          2012-07-22 13:02:28,246 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] catalog.MetaEditor(165): Added 1 regions in META
          2012-07-22 13:02:28,246 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(941): Closing tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.: disabling compactions & flushes
          2012-07-22 13:02:28,246 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(962): Updates disabled for region tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.
          2012-07-22 13:02:28,246 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(1010): Closed tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.
          2012-07-22 13:02:28,247 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0.logSyncer] wal.HLog$LogSyncer(1237): MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0.logSyncer exiting
          2012-07-22 13:02:28,247 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/tableA/a697a37d317a95f9881b48cc409eafb1/.logs
          2012-07-22 13:02:28,285 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/tableA/a697a37d317a95f9881b48cc409eafb1/.oldlogs
          2012-07-22 13:02:28,301 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager(2354): Bulk assigning 1 region(s) round-robin across 1 server(s)
          2012-07-22 13:02:28,302 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager$StartupBulkAssigner(2496): Timeout-on-RIT=1000
          2012-07-22 13:02:28,302 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1438): Bulk assigning 1 region(s) to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,303 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager(2359): Bulk assigning done
          2012-07-22 13:02:28,304 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] zookeeper.ZKAssign(177): master:55921-0x138aec80e530000 Async create of unassigned node for a697a37d317a95f9881b48cc409eafb1 with OFFLINE state
          2012-07-22 13:02:28,321 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:28,323 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(236): master:55921-0x138aec80e530000 Set watcher on existing znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1
          2012-07-22 13:02:28,324 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager$CreateUnassignedAsyncCallback(1603): rs=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. state=OFFLINE, ts=1342962148304, server=null, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,325 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager$ExistsUnassignedAsyncCallback(1634): rs=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. state=OFFLINE, ts=1342962148304, server=null
          2012-07-22 13:02:28,326 INFO [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1469): vesta.apache.org,56523,1342962143399 unassigned znodes=1 of total=1
          2012-07-22 13:02:28,326 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,327 INFO [PRI IPC Server handler 4 on 56523] regionserver.HRegionServer(3428): Received request to open region: tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,333 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node a697a37d317a95f9881b48cc409eafb1 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:28,334 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1527): Bulk assigning done for vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,334 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1; data=region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962148304, payload.length=0
          2012-07-22 13:02:28,343 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/a697a37d317a95f9881b48cc409eafb1
          2012-07-22 13:02:28,343 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node a697a37d317a95f9881b48cc409eafb1 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:28,343 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] regionserver.HRegion(3851): Opening region:

          Unknown macro: {NAME => 'tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.', STARTKEY => '', ENDKEY => '', ENCODED => a697a37d317a95f9881b48cc409eafb1,}

          2012-07-22 13:02:28,343 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1 and set watcher; region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148335, payload.length=0
          2012-07-22 13:02:28,343 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=a697a37d317a95f9881b48cc409eafb1, current state from RIT=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. state=PENDING_OPEN, ts=1342962148325, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,343 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] regionserver.HRegion(464): Instantiated tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.
          2012-07-22 13:02:28,345 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/tableA/a697a37d317a95f9881b48cc409eafb1/.tmp/.regioninfo with permission=rwxrwxrwx
          2012-07-22 13:02:28,415 INFO [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] regionserver.HRegion(609): Onlined tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1.; next sequenceid=0
          2012-07-22 13:02:28,415 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node a697a37d317a95f9881b48cc409eafb1 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:28,416 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1; data=region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148335, payload.length=0
          2012-07-22 13:02:28,420 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/a697a37d317a95f9881b48cc409eafb1
          2012-07-22 13:02:28,420 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node a697a37d317a95f9881b48cc409eafb1 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:28,420 INFO [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] regionserver.HRegionServer(1644): Post open deploy tasks for region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., daughter=false
          2012-07-22 13:02:28,421 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1 and set watcher; region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148416, payload.length=0
          2012-07-22 13:02:28,421 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=a697a37d317a95f9881b48cc409eafb1, current state from RIT=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. state=OPENING, ts=1342962148335, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,421 DEBUG [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] client.HConnectionManager$HConnectionImplementation(938): Looking up root region location in ZK, connection=hconnection 0xe8f27e
          2012-07-22 13:02:28,421 DEBUG [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] zookeeper.ZKUtil(1142): hconnection 0xe8f27e-0x138aec80e530003 Retrieved 35 byte(s) of data from znode /hbase/root-region-server; data=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,422 DEBUG [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] client.HConnectionManager$HConnectionImplementation(943): Looked up root region location, connection=hconnection 0xe8f27e; serverName=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,423 DEBUG [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] client.HConnectionManager$HConnectionImplementation(1335): Cached location for .META.,,1.1028785192 is vesta.apache.org:56523
          2012-07-22 13:02:28,427 INFO [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] catalog.MetaEditor(263): Updated row tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. with server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,428 INFO [PostOpenDeployTasks:a697a37d317a95f9881b48cc409eafb1] regionserver.HRegionServer(1669): Done with post open deploy task for region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., daughter=false
          2012-07-22 13:02:28,428 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node a697a37d317a95f9881b48cc409eafb1 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:28,429 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1; data=region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148416, payload.length=0
          2012-07-22 13:02:28,438 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/a697a37d317a95f9881b48cc409eafb1
          2012-07-22 13:02:28,438 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node a697a37d317a95f9881b48cc409eafb1 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:28,438 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] handler.OpenRegionHandler(286): region transitioned to opened in zookeeper:

          , server: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,458 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-1] handler.OpenRegionHandler(142): Opened tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. on server:vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,458 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1 and set watcher; region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148429, payload.length=0
          2012-07-22 13:02:28,459 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENED, server=vesta.apache.org,56523,1342962143399, region=a697a37d317a95f9881b48cc409eafb1, current state from RIT=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. state=OPENING, ts=1342962148416, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,459 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-2] handler.OpenedRegionHandler(149): Handling OPENED event for tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. from vesta.apache.org,56523,1342962143399; deleting unassigned node
          2012-07-22 13:02:28,459 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-2] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for a697a37d317a95f9881b48cc409eafb1 that is in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:28,460 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-2] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/a697a37d317a95f9881b48cc409eafb1; data=region=tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1., state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962148429, payload.length=0
          2012-07-22 13:02:28,466 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/a697a37d317a95f9881b48cc409eafb1
          2012-07-22 13:02:28,466 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(1191): The znode of region tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. has been deleted.
          2012-07-22 13:02:28,466 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-2] zookeeper.ZKAssign(521): master:55921-0x138aec80e530000 Successfully deleted unassigned node for region a697a37d317a95f9881b48cc409eafb1 in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:28,466 INFO [pool-1-thread-1-EventThread] master.AssignmentManager(1204): The master has opened the region tableA,,1342962147936.a697a37d317a95f9881b48cc409eafb1. that was online on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:28,467 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:29,128 DEBUG [Thread-137] client.MetaScanner(202): Scanning .META. starting at row=tableA,,00000000000000 for max=2147483647 rows using hconnection 0x1fd9b49
          2012-07-22 13:02:29,153 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 6 byte(s) of data from znode /hbase/table/tableA; data=PBUF\x08\x00
          2012-07-22 13:02:29,154 DEBUG [Thread-137] zookeeper.ZKUtil(103): catalogtracker-on-hconnection 0x1fd9b49 opening connection to ZooKeeper with ensemble (localhost:59288)
          2012-07-22 13:02:29,155 INFO [Thread-137] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org
          2012-07-22 13:02:29,155 DEBUG [Thread-137] catalog.CatalogTracker(240): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@14e5d57
          2012-07-22 13:02:29,164 DEBUG [Thread-137-EventThread] zookeeper.ZooKeeperWatcher(262): catalogtracker-on-hconnection 0x1fd9b49 Received ZooKeeper Event, type=None, state=SyncConnected, path=null
          2012-07-22 13:02:29,164 DEBUG [Thread-137] zookeeper.ZKUtil(236): catalogtracker-on-hconnection 0x1fd9b49 Set watcher on existing znode /hbase/root-region-server
          2012-07-22 13:02:29,164 DEBUG [Thread-137] zookeeper.ZKUtil(1142): catalogtracker-on-hconnection 0x1fd9b49 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,165 DEBUG [Thread-137] zookeeper.ZKUtil(238): catalogtracker-on-hconnection 0x1fd9b49 /hbase/unassigned/1028785192 does not exist. Watcher is set.
          2012-07-22 13:02:29,165 DEBUG [Thread-137-EventThread] zookeeper.ZooKeeperWatcher(339): catalogtracker-on-hconnection 0x1fd9b49-0x138aec80e530006 connected
          2012-07-22 13:02:29,165 DEBUG [Thread-137] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableA,,'
          2012-07-22 13:02:29,165 DEBUG [Thread-137] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableA,,'
          2012-07-22 13:02:29,169 DEBUG [Thread-137] catalog.CatalogTracker(257): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@14e5d57
          2012-07-22 13:02:29,179 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableB,,'
          2012-07-22 13:02:29,179 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableB,,'
          2012-07-22 13:02:29,181 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(192): Finished with scanning at

          Unknown macro: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}

          2012-07-22 13:02:29,200 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.CreateTableHandler(128): Attempting to create the table tableB
          2012-07-22 13:02:29,200 DEBUG [Thread-137] client.MetaScanner(202): Scanning .META. starting at row=tableB,,00000000000000 for max=2147483647 rows using hconnection 0x1fd9b49
          2012-07-22 13:02:29,201 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] util.FSUtils(1114): hdfs://localhost:52085/user/hudson/hbase/tableB doesn't exist
          2012-07-22 13:02:29,245 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(3781): creating HRegion tableB HTD == 'tableB' RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == tableB
          2012-07-22 13:02:29,293 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms
          2012-07-22 13:02:29,309 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840
          2012-07-22 13:02:29,309 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26/.logs/hlog.1342962149293, compression=false
          2012-07-22 13:02:29,309 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(652): for /user/hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26/.logs/hlog.1342962149293
          2012-07-22 13:02:29,309 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(500): Using getNumCurrentReplicas--HDFS-826
          2012-07-22 13:02:29,310 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(464): Instantiated tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:29,312 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] catalog.MetaEditor(165): Added 1 regions in META
          2012-07-22 13:02:29,313 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(941): Closing tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.: disabling compactions & flushes
          2012-07-22 13:02:29,313 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(962): Updates disabled for region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:29,313 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] regionserver.HRegion(1010): Closed tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:29,313 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0.logSyncer] wal.HLog$LogSyncer(1237): MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0.logSyncer exiting
          2012-07-22 13:02:29,313 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26/.logs
          2012-07-22 13:02:29,344 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26/.oldlogs
          2012-07-22 13:02:29,356 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager(2354): Bulk assigning 1 region(s) round-robin across 1 server(s)
          2012-07-22 13:02:29,357 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager$StartupBulkAssigner(2496): Timeout-on-RIT=1000
          2012-07-22 13:02:29,357 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1438): Bulk assigning 1 region(s) to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,357 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager(2359): Bulk assigning done
          2012-07-22 13:02:29,357 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] zookeeper.ZKAssign(177): master:55921-0x138aec80e530000 Async create of unassigned node for e9d1ea1a8999939667d847817f3fba26 with OFFLINE state
          2012-07-22 13:02:29,363 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:29,371 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(236): master:55921-0x138aec80e530000 Set watcher on existing znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:29,371 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager$CreateUnassignedAsyncCallback(1603): rs=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=OFFLINE, ts=1342962149357, server=null, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,372 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager$ExistsUnassignedAsyncCallback(1634): rs=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=OFFLINE, ts=1342962149357, server=null
          2012-07-22 13:02:29,372 INFO [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1469): vesta.apache.org,56523,1342962143399 unassigned znodes=1 of total=1
          2012-07-22 13:02:29,372 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,374 INFO [PRI IPC Server handler 6 on 56523] regionserver.HRegionServer(3428): Received request to open region: tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,380 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node e9d1ea1a8999939667d847817f3fba26 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:29,380 DEBUG [vesta.apache.org,55921,1342962143091-StartupBulkAssigner-0] master.AssignmentManager(1527): Bulk assigning done for vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,381 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962149357, payload.length=0
          2012-07-22 13:02:29,391 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:29,391 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node e9d1ea1a8999939667d847817f3fba26 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
          2012-07-22 13:02:29,391 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] regionserver.HRegion(3851): Opening region:

          Unknown macro: {NAME => 'tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.', STARTKEY => '', ENDKEY => '', ENCODED => e9d1ea1a8999939667d847817f3fba26,}

          2012-07-22 13:02:29,391 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26 and set watcher; region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149381, payload.length=0
          2012-07-22 13:02:29,392 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=e9d1ea1a8999939667d847817f3fba26, current state from RIT=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=PENDING_OPEN, ts=1342962149372, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,392 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] regionserver.HRegion(464): Instantiated tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:29,393 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26/.tmp/.regioninfo with permission=rwxrwxrwx
          2012-07-22 13:02:29,454 INFO [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] regionserver.HRegion(609): Onlined tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.; next sequenceid=0
          2012-07-22 13:02:29,454 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node e9d1ea1a8999939667d847817f3fba26 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:29,455 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149381, payload.length=0
          2012-07-22 13:02:29,458 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:29,458 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node e9d1ea1a8999939667d847817f3fba26 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
          2012-07-22 13:02:29,459 INFO [PostOpenDeployTasks:e9d1ea1a8999939667d847817f3fba26] regionserver.HRegionServer(1644): Post open deploy tasks for region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., daughter=false
          2012-07-22 13:02:29,459 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26 and set watcher; region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149455, payload.length=0
          2012-07-22 13:02:29,459 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=e9d1ea1a8999939667d847817f3fba26, current state from RIT=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=OPENING, ts=1342962149381, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,462 INFO [PostOpenDeployTasks:e9d1ea1a8999939667d847817f3fba26] catalog.MetaEditor(263): Updated row tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. with server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,462 INFO [PostOpenDeployTasks:e9d1ea1a8999939667d847817f3fba26] regionserver.HRegionServer(1669): Done with post open deploy task for region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., daughter=false
          2012-07-22 13:02:29,463 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node e9d1ea1a8999939667d847817f3fba26 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:29,463 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149455, payload.length=0
          2012-07-22 13:02:29,470 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:29,470 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node e9d1ea1a8999939667d847817f3fba26 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
          2012-07-22 13:02:29,471 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] handler.OpenRegionHandler(286): region transitioned to opened in zookeeper:

          , server: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,471 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-2] handler.OpenRegionHandler(142): Opened tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. on server:vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,471 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26 and set watcher; region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149464, payload.length=0
          2012-07-22 13:02:29,471 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENED, server=vesta.apache.org,56523,1342962143399, region=e9d1ea1a8999939667d847817f3fba26, current state from RIT=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=OPENING, ts=1342962149455, server=vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,472 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-3] handler.OpenedRegionHandler(149): Handling OPENED event for tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. from vesta.apache.org,56523,1342962143399; deleting unassigned node
          2012-07-22 13:02:29,472 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-3] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for e9d1ea1a8999939667d847817f3fba26 that is in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:29,473 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-3] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962149464, payload.length=0
          2012-07-22 13:02:29,482 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:29,482 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(1191): The znode of region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. has been deleted.
          2012-07-22 13:02:29,482 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-3] zookeeper.ZKAssign(521): master:55921-0x138aec80e530000 Successfully deleted unassigned node for region e9d1ea1a8999939667d847817f3fba26 in expected state RS_ZK_REGION_OPENED
          2012-07-22 13:02:29,482 INFO [pool-1-thread-1-EventThread] master.AssignmentManager(1204): The master has opened the region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. that was online on vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:29,483 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:30,203 DEBUG [Thread-137] client.MetaScanner(202): Scanning .META. starting at row=tableB,,00000000000000 for max=2147483647 rows using hconnection 0x1fd9b49
          2012-07-22 13:02:30,223 INFO [Thread-137] client.HBaseAdmin$5(788): Started disable of tableB
          2012-07-22 13:02:30,224 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableB,,'
          2012-07-22 13:02:30,225 DEBUG [IPC Server handler 4 on 55921] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableB,,'
          2012-07-22 13:02:30,234 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.DisableTableHandler(101): Attempting to disable table tableB
          2012-07-22 13:02:30,243 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 6 byte(s) of data from znode /hbase/table/tableB; data=PBUF\x08\x02
          2012-07-22 13:02:30,243 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.DisableTableHandler(133): Offlining 1 regions.
          2012-07-22 13:02:30,243 DEBUG [Thread-137] client.HBaseAdmin(824): Sleeping= 1000ms, waiting for all regions to be disabled in tableB
          2012-07-22 13:02:30,244 DEBUG [vesta.apache.org,55921,1342962143091-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2065): Starting unassignment of region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. (offlining)
          2012-07-22 13:02:30,245 DEBUG [vesta.apache.org,55921,1342962143091-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(570): master:55921-0x138aec80e530000 Creating unassigned node for e9d1ea1a8999939667d847817f3fba26 in a CLOSING state
          2012-07-22 13:02:30,254 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:30,255 DEBUG [vesta.apache.org,55921,1342962143091-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:30,255 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(236): master:55921-0x138aec80e530000 Set watcher on existing znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:30,257 INFO [PRI IPC Server handler 7 on 56523] regionserver.HRegionServer(3494): Received close region: tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.. Version of ZK closing node:0. Destination server:null
          2012-07-22 13:02:30,258 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] handler.CloseRegionHandler(125): Processing close of tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,259 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(941): Closing tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.: disabling compactions & flushes
          2012-07-22 13:02:30,259 DEBUG [vesta.apache.org,55921,1342962143091-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2176): Sent CLOSE to vesta.apache.org,56523,1342962143399 for region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,259 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(962): Updates disabled for region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,259 INFO [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(1010): Closed tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,259 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node e9d1ea1a8999939667d847817f3fba26 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED
          2012-07-22 13:02:30,260 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 103 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=M_ZK_REGION_CLOSING, servername=vesta.apache.org,55921,1342962143091, createTime=1342962150245, payload.length=0
          2012-07-22 13:02:30,267 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:30,267 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node e9d1ea1a8999939667d847817f3fba26 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED
          2012-07-22 13:02:30,267 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] handler.CloseRegionHandler(160): set region closed state in zk successfully for region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. sn name: vesta.apache.org,56523,1342962143399
          2012-07-22 13:02:30,267 DEBUG [RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399-0] handler.CloseRegionHandler(169): Closed region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,267 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26 and set watcher; region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_CLOSED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962150260, payload.length=0
          2012-07-22 13:02:30,268 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_CLOSED, server=vesta.apache.org,56523,1342962143399, region=e9d1ea1a8999939667d847817f3fba26, current state from RIT=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. state=PENDING_CLOSE, ts=1342962150255, server=null
          2012-07-22 13:02:30,269 DEBUG [MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091-0] handler.ClosedRegionHandler(94): Handling CLOSED event for e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:30,269 DEBUG [MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091-0] master.AssignmentManager(1356): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26.
          2012-07-22 13:02:30,269 DEBUG [MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for e9d1ea1a8999939667d847817f3fba26 that is in expected state RS_ZK_REGION_CLOSED
          2012-07-22 13:02:30,270 DEBUG [MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 101 byte(s) of data from znode /hbase/unassigned/e9d1ea1a8999939667d847817f3fba26; data=region=tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26., state=RS_ZK_REGION_CLOSED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962150260, payload.length=0
          2012-07-22 13:02:30,279 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:30,279 DEBUG [MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKAssign(521): master:55921-0x138aec80e530000 Successfully deleted unassigned node for region e9d1ea1a8999939667d847817f3fba26 in expected state RS_ZK_REGION_CLOSED
          2012-07-22 13:02:30,279 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(1191): The znode of region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. has been deleted.
          2012-07-22 13:02:30,279 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned
          2012-07-22 13:02:30,299 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.DisableTableHandler(149): Disabled table is done=true
          2012-07-22 13:02:31,244 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 6 byte(s) of data from znode /hbase/table/tableB; data=PBUF\x08\x01
          2012-07-22 13:02:31,244 INFO [Thread-137] client.HBaseAdmin(840): Disabled tableB
          2012-07-22 13:02:31,245 DEBUG [Thread-137] zookeeper.ZKUtil(1142): hconnection 0x1fd9b49-0x138aec80e530005 Retrieved 6 byte(s) of data from znode /hbase/table/tableB; data=PBUF\x08\x01
          2012-07-22 13:02:31,248 DEBUG [IPC Server handler 2 on 55921] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableB,,'
          2012-07-22 13:02:31,248 DEBUG [IPC Server handler 2 on 55921] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableB,,'
          2012-07-22 13:02:31,254 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.TableEventHandler(95): Handling table operation C_M_DELETE_TABLE on table tableB
          2012-07-22 13:02:31,254 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] client.ClientScanner(94): Creating scanner over .META. starting at key 'tableB,,'
          2012-07-22 13:02:31,254 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] client.ClientScanner(205): Advancing internal scanner to startKey at 'tableB,,'
          2012-07-22 13:02:31,258 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] client.ClientScanner(192): Finished with scanning at

          2012-07-22 13:02:31,258 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] handler.DeleteTableHandler(75): Deleting region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. from META and FS
          2012-07-22 13:02:31,264 INFO [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] catalog.MetaEditor(278): Deleted region tableB,,1342962149177.e9d1ea1a8999939667d847817f3fba26. from META
          2012-07-22 13:02:31,265 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] backup.HFileArchiver(91): ARCHIVING region file:/tmp/hbase-hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26
          2012-07-22 13:02:31,266 DEBUG [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] backup.HFileArchiver(109): Have an archive directory, preparing to move files
          2012-07-22 13:02:31,267 ERROR [MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091-0] executor.EventHandler(198): Caught throwable while processing event C_M_DELETE_TABLE
          java.lang.IllegalArgumentException: Wrong FS: file:/tmp/hbase-hudson/hbase/tableB/e9d1ea1a8999939667d847817f3fba26, expected: hdfs://localhost:52085
          at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:551)
          at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:166)
          at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:358)
          at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1279)
          at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1319)
          at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1111)
          at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:122)
          at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:72)

          Doesn't seem too bad to fix. I'll post another patch up that solves both.

          Show
          Jesse Yates added a comment - Hmm, looks like I broke the 2.0 build too. Here is the output from the logs there that seems pertinent: Regression org.apache.hadoop.hbase.avro.TestAvroServer.testTableAdminAndMetadata Failing for the past 1 build (Since Failed#103 ) Took 32 ms. Error Message test timed out after 300000 milliseconds Stacktrace java.lang.Exception: test timed out after 300000 milliseconds at $Proxy26.scan(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:548) at org.apache.hadoop.hbase.avro.AvroServer$HBaseImpl.deleteTable(AvroServer.java:276) at org.apache.hadoop.hbase.avro.TestAvroServer.testTableAdminAndMetadata(TestAvroServer.java:122) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62) Standard Output Formatting using clusterid: testClusterID Standard Error 2012-07-22 13:02:19,346 WARN [pool-1-thread-1] conf.Configuration(737): hadoop.native.lib is deprecated. Instead, use io.native.lib.available 2012-07-22 13:02:19,491 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(614): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2012-07-22 13:02:19,607 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(288): Created new mini-cluster data directory: /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce 2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting test.cache.data to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/cache_data in system properties and HBase conf 2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting hadoop.tmp.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/hadoop_tmp in system properties and HBase conf 2012-07-22 13:02:19,608 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting hadoop.log.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/hadoop_logs in system properties and HBase conf 2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapreduce.output.fileoutputformat.outputdir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_output in system properties and HBase conf 2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.local.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_local in system properties and HBase conf 2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.system.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_system in system properties and HBase conf 2012-07-22 13:02:19,616 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(466): Setting mapred.temp.dir to /home/hudson/hudson-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/test-data/03dca982-ed63-4680-a4c4-4a7034cab836/dfscluster_654272cb-8aec-4e5d-8796-0f03ff31fcce/mapred_temp in system properties and HBase conf 2012-07-22 13:02:19,846 WARN [pool-1-thread-1] conf.Configuration(737): hadoop.configured.node.mapping is deprecated. Instead, use net.topology.configured.node.mapping 2012-07-22 13:02:19,972 WARN [pool-1-thread-1] util.KerberosName(87): Kerberos krb5 configuration not found, setting default realm to empty 2012-07-22 13:02:20,673 WARN [pool-1-thread-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2012-07-22 13:02:21,523 INFO [pool-1-thread-1] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2012-07-22 13:02:21,578 WARN [pool-1-thread-1] lib.StaticUserWebFilter(141): dfs.web.ugi should not be used. Instead, use hadoop.http.staticuser.user. 2012-07-22 13:02:21,598 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26 2012-07-22 13:02:21,625 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar: file:/home/hudson/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.0-alpha/hadoop-hdfs-2.0.0-alpha-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_39374_hdfs____.ksh5ik/webapp 2012-07-22 13:02:21,777 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:39374 2012-07-22 13:02:21,830 WARN [pool-1-thread-1] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2012-07-22 13:02:21,848 WARN [pool-1-thread-1] lib.StaticUserWebFilter(141): dfs.web.ugi should not be used. Instead, use hadoop.http.staticuser.user. 2012-07-22 13:02:21,851 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26 2012-07-22 13:02:21,854 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar: file:/home/hudson/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.0-alpha/hadoop-hdfs-2.0.0-alpha-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_47954_datanode____.oum544/webapp 2012-07-22 13:02:21,933 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:47954 2012-07-22 13:02:22,755 WARN [pool-1-thread-1] conf.Configuration(737): fs.default.name is deprecated. Instead, use fs.defaultFS 2012-07-22 13:02:22,827 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(197): Started MiniZK Cluster and connect 1 ZK server on client port: 59288 2012-07-22 13:02:22,978 DEBUG [pool-1-thread-1] util.FSUtils(438): Created version file at hdfs://localhost:52085/user/hudson/hbase with version=7 2012-07-22 13:02:23,026 DEBUG [pool-1-thread-1] client.HConnectionManager(2483): Set serverside HConnection retries=100 2012-07-22 13:02:23,050 INFO [pool-1-thread-1] ipc.HBaseRpcMetrics(66): Initializing RPC Metrics with hostName=HMaster, port=55921 2012-07-22 13:02:23,095 WARN [pool-1-thread-1] conf.Configuration(737): mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 2012-07-22 13:02:23,102 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(103): master:55921 opening connection to ZooKeeper with ensemble (localhost:59288) 2012-07-22 13:02:23,122 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org 2012-07-22 13:02:23,238 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-07-22 13:02:23,242 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(339): master:55921-0x138aec80e530000 connected 2012-07-22 13:02:23,322 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: revision 2012-07-22 13:02:23,322 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsUser 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsDate 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsUrl 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: date 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsRevision 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: user 2012-07-22 13:02:23,323 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: hdfsVersion 2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: url 2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(110): MetricsString added: version 2012-07-22 13:02:23,324 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(161): new MBeanInfo 2012-07-22 13:02:23,324 ERROR [pool-1-thread-1] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:23,325 ERROR [pool-1-thread-1] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:23,325 INFO [pool-1-thread-1] metrics.MetricsMBeanBase(161): new MBeanInfo 2012-07-22 13:02:23,325 INFO [pool-1-thread-1] metrics.MasterMetrics(102): Initialized 2012-07-22 13:02:23,356 DEBUG [pool-1-thread-1] client.HConnectionManager(2483): Set serverside HConnection retries=100 2012-07-22 13:02:23,359 INFO [pool-1-thread-1] ipc.HBaseRpcMetrics(66): Initializing RPC Metrics with hostName=MiniHBaseCluster$MiniHBaseClusterRegionServer, port=56523 2012-07-22 13:02:23,401 INFO [pool-1-thread-1] hfile.CacheConfig(350): Allocating LruBlockCache with maximum size 422.2m 2012-07-22 13:02:23,411 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/shutdown does not exist. Watcher is set. 2012-07-22 13:02:23,421 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ActiveMasterManager(150): Deleting ZNode for /hbase/backup-masters/vesta.apache.org,55921,1342962143091 from backup master directory 2012-07-22 13:02:23,431 WARN [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(142): Node /hbase/backup-masters/vesta.apache.org,55921,1342962143091 already deleted, and this is not a retry 2012-07-22 13:02:23,432 WARN [Master:0;vesta.apache.org,55921,1342962143091] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2012-07-22 13:02:23,432 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ActiveMasterManager(159): Master=vesta.apache.org,55921,1342962143091 2012-07-22 13:02:23,434 WARN [Master:0;vesta.apache.org,55921,1342962143091] conf.Configuration(737): fs.default.name is deprecated. Instead, use fs.defaultFS 2012-07-22 13:02:23,438 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(175): timeout = 25000 2012-07-22 13:02:23,438 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(176): unassigned timeout = 180000 2012-07-22 13:02:23,441 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.SplitLogManager(806): found 0 orphan tasks and 0 rescan nodes 2012-07-22 13:02:23,504 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(567): Created cluster ID file at hdfs://localhost:52085/user/hudson/hbase/hbase.id with ID: f285209f-4f02-4b1d-817f-d5611467793a 2012-07-22 13:02:23,508 INFO [pool-1-thread-1] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RegionServer:0;vesta.apache.org,56523,1342962143399 2012-07-22 13:02:23,512 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(103): regionserver:56523 opening connection to ZooKeeper with ensemble (localhost:59288) 2012-07-22 13:02:23,513 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org 2012-07-22 13:02:23,521 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-07-22 13:02:23,522 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(339): regionserver:56523-0x138aec80e530001 connected 2012-07-22 13:02:23,522 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(236): regionserver:56523-0x138aec80e530001 Set watcher on existing znode /hbase/master 2012-07-22 13:02:23,527 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 35 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A\x1D\x0A\x10vesta.apa... 2012-07-22 13:02:23,527 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/shutdown does not exist. Watcher is set. 2012-07-22 13:02:23,539 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.MasterFileSystem(394): BOOTSTRAP: creating ROOT and first META regions 2012-07-22 13:02:23,539 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(3781): creating HRegion ROOT HTD == ' ROOT ', Unknown macro: {METHOD => 'table_att', IS_META => 'true', IS_ROOT => 'true'} , Unknown macro: {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'false'} RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == ROOT 2012-07-22 13:02:23,580 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2012-07-22 13:02:23,622 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840 2012-07-22 13:02:23,622 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.logs/hlog.1342962143580, compression=false 2012-07-22 13:02:23,622 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(652): for /user/hudson/hbase/ ROOT /70236052/.logs/hlog.1342962143580 2012-07-22 13:02:23,623 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(500): Using getNumCurrentReplicas-- HDFS-826 2012-07-22 13:02:23,639 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(464): Instantiated ROOT ,,0.70236052 2012-07-22 13:02:23,643 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/.regioninfo with permission=rwxrwxrwx 2012-07-22 13:02:23,730 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(224): hbase.hstore.compaction.min = 3 2012-07-22 13:02:23,738 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] util.ChecksumType$2(68): Checksum using org.apache.hadoop.util.PureJavaCrc32 2012-07-22 13:02:23,738 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] util.ChecksumType$3(111): Checksum can use org.apache.hadoop.util.PureJavaCrc32C 2012-07-22 13:02:23,748 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(609): Onlined ROOT ,,0.70236052; next sequenceid=1 2012-07-22 13:02:23,748 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(3781): creating HRegion .META. HTD == '.META.', Unknown macro: {METHOD => 'table_att', IS_META => 'true'} , RootDir = hdfs://localhost:52085/user/hudson/hbase Table name == .META. 2012-07-22 13:02:23,787 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2012-07-22 13:02:23,803 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840 2012-07-22 13:02:23,803 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.logs/hlog.1342962143787, compression=false 2012-07-22 13:02:23,803 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(652): for /user/hudson/hbase/.META./1028785192/.logs/hlog.1342962143787 2012-07-22 13:02:23,803 INFO [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(500): Using getNumCurrentReplicas-- HDFS-826 2012-07-22 13:02:23,804 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(464): Instantiated .META.,,1.1028785192 2012-07-22 13:02:23,806 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.tmp/.regioninfo with permission=rwxrwxrwx 2012-07-22 13:02:23,882 INFO [StoreOpenerThread-.META.,,1.1028785192-1] regionserver.Store(224): hbase.hstore.compaction.min = 3 2012-07-22 13:02:23,887 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(609): Onlined .META.,,1.1028785192; next sequenceid=1 2012-07-22 13:02:23,909 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(941): Closing ROOT ,,0.70236052: disabling compactions & flushes 2012-07-22 13:02:23,909 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(962): Updates disabled for region ROOT ,,0.70236052 2012-07-22 13:02:23,910 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1468): Started memstore flush for ROOT ,,0.70236052, current region memstore size 360 2012-07-22 13:02:23,910 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1516): Finished snapshotting ROOT ,,0.70236052, commencing wait for mvcc, flushsize=360 2012-07-22 13:02:23,911 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1526): Finished snapshotting, commencing flushing stores 2012-07-22 13:02:23,932 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] util.FSUtils(163): Creating file=hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0 with permission=rwxrwxrwx 2012-07-22 13:02:23,953 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] hfile.HFileWriterV2(143): Initialized with CacheConfig:enabled [cacheDataOnRead=false] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false] 2012-07-22 13:02:23,956 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.StoreFile$Writer(1001): Delete Family Bloom filter type for hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0: CompoundBloomFilterWriter 2012-07-22 13:02:23,995 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.StoreFile$Writer(1221): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0) 2012-07-22 13:02:23,996 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(785): Flushed , sequenceid=2, memsize=360, into tmp file hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0 2012-07-22 13:02:24,013 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(810): Renaming flushed file at hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.tmp/12a0be11d2894ba0ab2cc64ec6357aa0 to hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0 2012-07-22 13:02:24,031 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.Store(833): Added hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0, entries=2, sequenceid=2, filesize=848 2012-07-22 13:02:24,032 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1600): Finished memstore flush of ~360/360, currentsize=0/0 for region ROOT ,,0.70236052 in 122ms, sequenceid=2, compaction requested=false 2012-07-22 13:02:24,035 INFO [StoreCloserThread--ROOT-,,0.70236052-1] regionserver.Store(670): Closed info 2012-07-22 13:02:24,035 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1010): Closed ROOT ,,0.70236052 2012-07-22 13:02:24,035 INFO [Master:0;vesta.apache.org,55921,1342962143091.logSyncer] wal.HLog$LogSyncer(1237): Master:0;vesta.apache.org,55921,1342962143091.logSyncer exiting 2012-07-22 13:02:24,036 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/.logs 2012-07-22 13:02:24,061 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/ ROOT /70236052/.oldlogs 2012-07-22 13:02:24,073 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(941): Closing .META.,,1.1028785192: disabling compactions & flushes 2012-07-22 13:02:24,074 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(962): Updates disabled for region .META.,,1.1028785192 2012-07-22 13:02:24,074 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.Store(670): Closed info 2012-07-22 13:02:24,074 INFO [Master:0;vesta.apache.org,55921,1342962143091] regionserver.HRegion(1010): Closed .META.,,1.1028785192 2012-07-22 13:02:24,074 INFO [Master:0;vesta.apache.org,55921,1342962143091.logSyncer] wal.HLog$LogSyncer(1237): Master:0;vesta.apache.org,55921,1342962143091.logSyncer exiting 2012-07-22 13:02:24,075 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(1006): closing hlog writer in hdfs://localhost:52085/user/hudson/hbase/.META./1028785192/.logs 2012-07-22 13:02:24,109 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] wal.HLog(974): Moved 1 log files to /user/hudson/hbase/.META./1028785192/.oldlogs 2012-07-22 13:02:24,619 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(103): hconnection 0xcf7fd0 opening connection to ZooKeeper with ensemble (localhost:59288) 2012-07-22 13:02:24,620 INFO [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org 2012-07-22 13:02:24,629 DEBUG [Master:0;vesta.apache.org,55921,1342962143091-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0xcf7fd0 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-07-22 13:02:24,630 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(1142): hconnection 0xcf7fd0 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8... 2012-07-22 13:02:24,630 INFO [Master:0;vesta.apache.org,55921,1342962143091] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a 2012-07-22 13:02:24,630 DEBUG [Master:0;vesta.apache.org,55921,1342962143091-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0xcf7fd0-0x138aec80e530002 connected 2012-07-22 13:02:24,632 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] catalog.CatalogTracker(240): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@cf5006 2012-07-22 13:02:24,633 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/root-region-server does not exist. Watcher is set. 2012-07-22 13:02:24,633 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKUtil(238): master:55921-0x138aec80e530000 /hbase/unassigned/1028785192 does not exist. Watcher is set. 2012-07-22 13:02:24,654 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/shutdown 2012-07-22 13:02:24,654 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/shutdown 2012-07-22 13:02:24,655 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(524): Server active/primary master; vesta.apache.org,55921,1342962143091, sessionid=0x138aec80e530000, cluster-up flag was=false 2012-07-22 13:02:24,655 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 34 byte(s) of data from znode /hbase/shutdown and set watcher; PBUF\x0A\x1CSun Jul 22 13:02:... 2012-07-22 13:02:24,655 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 34 byte(s) of data from znode /hbase/shutdown and set watcher; PBUF\x0A\x1CSun Jul 22 13:02:... 2012-07-22 13:02:24,659 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(103): hconnection 0xe8f27e opening connection to ZooKeeper with ensemble (localhost:59288) 2012-07-22 13:02:24,660 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.RecoverableZooKeeper(102): The identifier of this process is 4689@vesta.apache.org 2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5 2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_CLOSE_REGION-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5 2012-07-22 13:02:24,662 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_SERVER_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=3, maxPoolSize=3 2012-07-22 13:02:24,663 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_META_SERVER_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=5, maxPoolSize=5 2012-07-22 13:02:24,663 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] executor.ExecutorService(133): Starting executor service name=MASTER_TABLE_OPERATIONS-vesta.apache.org,55921,1342962143091, corePoolSize=1, maxPoolSize=1 2012-07-22 13:02:24,667 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.HMaster(1034): Started service threads 2012-07-22 13:02:24,667 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(627): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2012-07-22 13:02:24,669 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): hconnection 0xe8f27e Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(1142): hconnection 0xe8f27e Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$f285209f-4f02-4b1d-8... 2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(339): hconnection 0xe8f27e-0x138aec80e530003 connected 2012-07-22 13:02:24,670 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] client.HConnectionManager$HConnectionImplementation(664): ClusterId is f285209f-4f02-4b1d-817f-d5611467793a 2012-07-22 13:02:24,670 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] catalog.CatalogTracker(240): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5b701a 2012-07-22 13:02:24,671 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/root-region-server does not exist. Watcher is set. 2012-07-22 13:02:24,672 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] zookeeper.ZKUtil(238): regionserver:56523-0x138aec80e530001 /hbase/unassigned/1028785192 does not exist. Watcher is set. 2012-07-22 13:02:24,674 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.MemStoreFlusher(112): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g 2012-07-22 13:02:24,676 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer$CompactionChecker(1216): Runs every 16mins, 40sec 2012-07-22 13:02:24,709 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1828): Attempting connect to Master server at vesta.apache.org,55921,1342962143091 2012-07-22 13:02:24,755 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1837): Connected to master at vesta.apache.org/67.195.138.9:55921 2012-07-22 13:02:24,755 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1883): Telling master at vesta.apache.org,55921,1342962143091 that we are up with port=56523, startcode=1342962143399 2012-07-22 13:02:24,759 INFO [IPC Server handler 2 on 55921] master.ServerManager(270): Registering server=vesta.apache.org,56523,1342962143399 2012-07-22 13:02:24,764 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1063): Config from master: hbase.rootdir=hdfs://localhost:52085/user/hudson/hbase 2012-07-22 13:02:24,764 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1063): Config from master: fs.default.name=hdfs://localhost:52085 2012-07-22 13:02:24,764 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1056): Master passed us hostname to use. Was=vesta.apache.org, Now=vesta.apache.org 2012-07-22 13:02:24,767 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(644): Finished waiting for region servers count to settle; checked in 1, slept for 100 ms, expecting minimum of 1, maximum of 1, master is running. 2012-07-22 13:02:24,768 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2012-07-22 13:02:24,768 WARN [RegionServer:0;vesta.apache.org,56523,1342962143399] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2012-07-22 13:02:24,769 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(236): master:55921-0x138aec80e530000 Set watcher on existing znode /hbase/rs/vesta.apache.org,56523,1342962143399 2012-07-22 13:02:24,772 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1277): logdir=hdfs://localhost:52085/user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399 2012-07-22 13:02:24,786 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(428): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2012-07-22 13:02:24,804 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.SequenceFileLogWriter(190): using new createWriter – HADOOP-6840 2012-07-22 13:02:24,804 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.SequenceFileLogWriter(194): Path=hdfs://localhost:52085/user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399/vesta.apache.org%2C56523%2C1342962143399.1342962144786, compression=false 2012-07-22 13:02:24,804 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(652): for /user/hudson/hbase/.logs/vesta.apache.org,56523,1342962143399/vesta.apache.org%2C56523%2C1342962143399.1342962144786 2012-07-22 13:02:24,804 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] wal.HLog(500): Using getNumCurrentReplicas-- HDFS-826 2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,807 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,808 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,808 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] util.MetricsDynamicMBeanBase(115): unknown metrics type: org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram 2012-07-22 13:02:24,808 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.MetricsMBeanBase(161): new MBeanInfo 2012-07-22 13:02:24,808 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.RegionServerMetrics(326): Initialized 2012-07-22 13:02:24,809 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.MetricsMBeanBase(161): new MBeanInfo 2012-07-22 13:02:24,810 ERROR [RegionServer:0;vesta.apache.org,56523,1342962143399] metrics.RegionServerDynamicMetrics(91): java.lang.NoSuchMethodException: org.apache.hadoop.hbase.metrics.MetricsMBeanBase.updateMbeanInfoIfMetricsListChanged() 2012-07-22 13:02:24,810 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_REGION-vesta.apache.org,56523,1342962143399, corePoolSize=3, maxPoolSize=3 2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_ROOT-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1 2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_OPEN_META-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1 2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_REGION-vesta.apache.org,56523,1342962143399, corePoolSize=3, maxPoolSize=3 2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_ROOT-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1 2012-07-22 13:02:24,811 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399] executor.ExecutorService(133): Starting executor service name=RS_CLOSE_META-vesta.apache.org,56523,1342962143399, corePoolSize=1, maxPoolSize=1 2012-07-22 13:02:24,819 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(1095): Serving as vesta.apache.org,56523,1342962143399, RPC listening on vesta.apache.org/67.195.138.9:56523, sessionid=0x138aec80e530001 2012-07-22 13:02:24,819 INFO [SplitLogWorker-vesta.apache.org,56523,1342962143399] regionserver.SplitLogWorker(135): SplitLogWorker vesta.apache.org,56523,1342962143399 starting 2012-07-22 13:02:24,820 INFO [RegionServer:0;vesta.apache.org,56523,1342962143399] regionserver.HRegionServer(2353): Registered RegionServer MXBean 2012-07-22 13:02:25,778 INFO [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RootRegionTracker(157): Unsetting ROOT region location in ZooKeeper 2012-07-22 13:02:25,783 WARN [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.RecoverableZooKeeper(142): Node /hbase/root-region-server already deleted, and this is not a retry 2012-07-22 13:02:25,785 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] zookeeper.ZKAssign(272): master:55921-0x138aec80e530000 Creating (or updating) unassigned node for 70236052 with OFFLINE state 2012-07-22 13:02:25,799 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1732): Setting table ROOT to ENABLED state. 2012-07-22 13:02:25,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1976): No previous transition plan was found (or we are ignoring an existing plan) for ROOT ,,0.70236052 so generated a random one; hri= ROOT ,,0.70236052, src=, dest=vesta.apache.org,56523,1342962143399; 1 (online=1, available=1) available servers 2012-07-22 13:02:25,819 INFO [Master:0;vesta.apache.org,55921,1342962143091] master.AssignmentManager(1751): Assigning region ROOT ,,0.70236052 to vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,819 DEBUG [Master:0;vesta.apache.org,55921,1342962143091] master.ServerManager(574): New connection to vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,846 INFO [PRI IPC Server handler 2 on 56523] regionserver.HRegionServer(3428): Received request to open region: ROOT ,,0.70236052 on vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,848 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ ROOT from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2012-07-22 13:02:25,849 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 57 byte(s) of data from znode /hbase/unassigned/70236052; data=region= ROOT ,,0, state=M_ZK_REGION_OFFLINE, servername=vesta.apache.org,55921,1342962143091, createTime=1342962145788, payload.length=0 2012-07-22 13:02:25,854 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052 2012-07-22 13:02:25,854 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2012-07-22 13:02:25,855 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(3851): Opening region: Unknown macro: {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,} 2012-07-22 13:02:25,855 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region= ROOT ,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145849, payload.length=0 2012-07-22 13:02:25,856 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=70236052/ ROOT , current state from RIT= ROOT ,,0.70236052 state=PENDING_OPEN, ts=1342962145819, server=vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,860 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(464): Instantiated ROOT ,,0.70236052 2012-07-22 13:02:25,900 INFO [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(224): hbase.hstore.compaction.min = 3 2012-07-22 13:02:25,912 DEBUG [StoreOpenerThread--ROOT-,,0.70236052-1] regionserver.Store(428): loaded hdfs://localhost:52085/user/hudson/hbase/ ROOT /70236052/info/12a0be11d2894ba0ab2cc64ec6357aa0, isReference=false, isBulkLoadResult=false, seqid=2, majorCompaction=false 2012-07-22 13:02:25,915 INFO [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] regionserver.HRegion(609): Onlined ROOT ,,0.70236052; next sequenceid=3 2012-07-22 13:02:25,916 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ ROOT from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2012-07-22 13:02:25,916 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=region= ROOT ,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145849, payload.length=0 2012-07-22 13:02:25,922 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052 2012-07-22 13:02:25,922 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2012-07-22 13:02:25,923 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region= ROOT ,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145917, payload.length=0 2012-07-22 13:02:25,923 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENING, server=vesta.apache.org,56523,1342962143399, region=70236052/ ROOT , current state from RIT= ROOT ,,0.70236052 state=OPENING, ts=1342962145849, server=vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,923 INFO [PostOpenDeployTasks:70236052] regionserver.HRegionServer(1644): Post open deploy tasks for region= ROOT ,,0.70236052, daughter=false 2012-07-22 13:02:25,923 INFO [PostOpenDeployTasks:70236052] zookeeper.RootRegionTracker(123): Setting ROOT region location in ZooKeeper as vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,932 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZooKeeperWatcher(262): regionserver:56523-0x138aec80e530001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/root-region-server 2012-07-22 13:02:25,932 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/root-region-server 2012-07-22 13:02:25,933 DEBUG [RegionServer:0;vesta.apache.org,56523,1342962143399-EventThread] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,933 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 35 byte(s) of data from znode /hbase/root-region-server and set watcher; vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,933 INFO [PostOpenDeployTasks:70236052] regionserver.HRegionServer(1669): Done with post open deploy task for region= ROOT ,,0.70236052, daughter=false 2012-07-22 13:02:25,934 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(763): regionserver:56523-0x138aec80e530001 Attempting to transition node 70236052/ ROOT from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2012-07-22 13:02:25,934 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKUtil(1142): regionserver:56523-0x138aec80e530001 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=region= ROOT ,,0, state=RS_ZK_REGION_OPENING, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145917, payload.length=0 2012-07-22 13:02:25,944 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(262): master:55921-0x138aec80e530000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/70236052 2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] zookeeper.ZKAssign(824): regionserver:56523-0x138aec80e530001 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(286): region transitioned to opened in zookeeper: , server: vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,944 DEBUG [RS_OPEN_REGION-vesta.apache.org,56523,1342962143399-0] handler.OpenRegionHandler(142): Opened ROOT ,,0.70236052 on server:vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,945 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052 and set watcher; region= ROOT ,,0, state=RS_ZK_REGION_OPENED, servername=vesta.apache.org,56523,1342962143399, createTime=1342962145935, payload.length=0 2012-07-22 13:02:25,945 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(773): Handling transition=RS_ZK_REGION_OPENED, server=vesta.apache.org,56523,1342962143399, region=70236052/ ROOT , current state from RIT= ROOT ,,0.70236052 state=OPENING, ts=1342962145917, server=vesta.apache.org,56523,1342962143399 2012-07-22 13:02:25,946 INFO [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] handler.OpenedRegionHandler(147): Handling OPENED event for ROOT ,,0.70236052 from vesta.apache.org,56523,1342962143399; deleting unassigned node 2012-07-22 13:02:25,946 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKAssign(492): master:55921-0x138aec80e530000 Deleting existing unassigned node for 70236052 that is in expected state RS_ZK_REGION_OPENED 2012-07-22 13:02:25,947 DEBUG [MASTER_OPEN_REGION-vesta.apache.org,55921,1342962143091-0] zookeeper.ZKUtil(1142): master:55921-0x138aec80e530000 Retrieved 55 byte(s) of data from znode /hbase/unassigned/70236052; data=