HBase
  1. HBase
  2. HBASE-451

Remove HTableDescriptor from HRegionInfo

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.92.0
    • Component/s: master, regionserver
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Removes HTableDescriptor from HRegionInfo. Runs a migration of catalog tables on startup.

      Description

      There is an HRegionInfo for every region in HBase. Currently HRegionInfo also contains the HTableDescriptor (the schema). That means we store the schema n times where n is the number of regions in the table.

      Additionally, for every region of the same table that the region server has open, there is a copy of the schema. Thus it is stored in memory once for each open region.

      If HRegionInfo merely contained the table name the HTableDescriptor could be stored in a separate file and easily found.

      1. 451_support_for_removing_HTD_from_HRI_trunk.txt
        221 kB
        Ted Yu
      2. 451-addendum-v2.txt
        1 kB
        Ted Yu
      3. descriptors.txt
        98 kB
        stack
      4. fixtestadmin.txt
        32 kB
        stack
      5. HBASE-451_-_First_draft_support_for_removing_HTD_from_HRI1.patch
        225 kB
        Subbu M Iyer
      6. HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch
        237 kB
        Subbu M Iyer
      7. HBASE-451_-_Second_draft_-_Remove_HTD_from_HRI.patch
        238 kB
        Subbu M Iyer
      8. HBASE-451-Fixed_broken_TestAdmin.patch
        8 kB
        Subbu M Iyer
      9. HBASE-451-Fixed_broken_TestAdmin1.patch
        2 kB
        Subbu M Iyer
      10. pass_htd_on_region_construction.txt
        45 kB
        stack

        Issue Links

          Activity

          Hide
          stack added a comment -

          How about a new catalog table named TABLE

          Show
          stack added a comment - How about a new catalog table named TABLE
          Hide
          Bryan Duxbury added a comment -

          I like that idea because then we don't have to work very hard to store that information. It also means we can check if a table exists by doing a single get, instead of a META scan. We can also quickly tell if META is corrupt because we have a reliable place to find table names.

          All in all, +1 to that idea.

          Show
          Bryan Duxbury added a comment - I like that idea because then we don't have to work very hard to store that information. It also means we can check if a table exists by doing a single get, instead of a META scan. We can also quickly tell if META is corrupt because we have a reliable place to find table names. All in all, +1 to that idea.
          Hide
          Jim Kellerman added a comment -

          I'd prefer that the HTableDescriptor be a file under /hbase-root/table-name. Why? So that the HBase-fsck can get all the info it needs from the file system if root or meta or the TABLE table were corrupted.

          Show
          Jim Kellerman added a comment - I'd prefer that the HTableDescriptor be a file under /hbase-root/table-name. Why? So that the HBase-fsck can get all the info it needs from the file system if root or meta or the TABLE table were corrupted.
          Hide
          Jim Kellerman added a comment -

          If we are going to have to modify HTableDescriptor to support user specified Comparators, seems like this would also be a good time to more HTableDescriptor out of HRegionInfo

          Show
          Jim Kellerman added a comment - If we are going to have to modify HTableDescriptor to support user specified Comparators, seems like this would also be a good time to more HTableDescriptor out of HRegionInfo
          Hide
          stack added a comment -

          Above issue talks about putting schema behind zookeeper interface

          Show
          stack added a comment - Above issue talks about putting schema behind zookeeper interface
          Hide
          Bryan Duxbury added a comment -

          If table descriptors lived in a .TABLE. table, would it be at the same level as user tables?

          Are there downsides to having a catalog table instead of storing in files? I'm not sure that the HBase-check thing is that big of an issue. The catalog table will almost always be pretty small, have a static name, and probably live in a pretty well known place. Even if META was corrupt, that wouldn't stop us from being able to recover the catalog table. (At least, not any more than it would stop us from recovering the rest of the user-level tables.)

          Show
          Bryan Duxbury added a comment - If table descriptors lived in a .TABLE. table, would it be at the same level as user tables? Are there downsides to having a catalog table instead of storing in files? I'm not sure that the HBase-check thing is that big of an issue. The catalog table will almost always be pretty small, have a static name, and probably live in a pretty well known place. Even if META was corrupt, that wouldn't stop us from being able to recover the catalog table. (At least, not any more than it would stop us from recovering the rest of the user-level tables.)
          Hide
          Jean-Daniel Cryans added a comment -

          This issue can be fixed using the HBase implementation of Zookeeper (see HBASE-546). What will be behind the interface can either be a new table TABLE or a file like Jim wrote. I prefer the file solution since it will work in the same way with ZK.

          Show
          Jean-Daniel Cryans added a comment - This issue can be fixed using the HBase implementation of Zookeeper (see HBASE-546 ). What will be behind the interface can either be a new table TABLE or a file like Jim wrote. I prefer the file solution since it will work in the same way with ZK.
          Hide
          stack added a comment -

          Thinking more on this – prompted by the J-D comment above – I agree that a hbase.rootdir/TABLEDIR/schema file – is the way to go rather than a new catalog table. The file will be read rarely – during the onlining/deploy of a region. The table schema file could be stored in ZK but having our schema in multiple places – in HDFS and in ZK – makes me queasy. The JK point above that if it were in the TABLEDIR directory, an offline hbasck repair tool would also have access, is a strong argument for keeping it in hdfs.

          Show
          stack added a comment - Thinking more on this – prompted by the J-D comment above – I agree that a hbase.rootdir/TABLEDIR/schema file – is the way to go rather than a new catalog table. The file will be read rarely – during the onlining/deploy of a region. The table schema file could be stored in ZK but having our schema in multiple places – in HDFS and in ZK – makes me queasy. The JK point above that if it were in the TABLEDIR directory, an offline hbasck repair tool would also have access, is a strong argument for keeping it in hdfs.
          Hide
          Jean-Daniel Cryans added a comment -

          IMO, writing the schema in HDFS or in Zookeeper's namespace should be the same operation so it should not be a major problem, can be the same code but with a different output. Regarding hbase-fsck, it could read from ZK or from HDFS depending on the configuration. Last point, ZK is supposed to be "read optimized" so reading from it when we are using it should be faster than HDFS.

          That said, we could just start by putting the schema in HDFS and see if it causes trouble and then we could consider putting it in ZK.

          Show
          Jean-Daniel Cryans added a comment - IMO, writing the schema in HDFS or in Zookeeper's namespace should be the same operation so it should not be a major problem, can be the same code but with a different output. Regarding hbase-fsck, it could read from ZK or from HDFS depending on the configuration. Last point, ZK is supposed to be "read optimized" so reading from it when we are using it should be faster than HDFS. That said, we could just start by putting the schema in HDFS and see if it causes trouble and then we could consider putting it in ZK.
          Hide
          stack added a comment -

          Agreed

          Show
          stack added a comment - Agreed
          Hide
          stack added a comment -

          I don't see any action on this issue. Should we move to 0.21?

          Show
          stack added a comment - I don't see any action on this issue. Should we move to 0.21?
          Hide
          stack added a comment -

          I don't think this will happen in the 0.20.0 timeframe. Moving it out.

          Show
          stack added a comment - I don't think this will happen in the 0.20.0 timeframe. Moving it out.
          Hide
          stack added a comment -

          Fix in 0.21.... moving schema to zk.

          Show
          stack added a comment - Fix in 0.21.... moving schema to zk.
          Hide
          stack added a comment -

          Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.

          Show
          stack added a comment - Moved from 0.21 to 0.22 just after merge of old 0.20 branch into TRUNK.
          Hide
          stack added a comment -
          Show
          stack added a comment - Design for this is up in http://wiki.apache.org/hadoop/Hbase/MasterRewrite
          Hide
          stack added a comment -

          Making it critical – this is about making it so can we can do schema edits online.

          Show
          stack added a comment - Making it critical – this is about making it so can we can do schema edits online.
          Hide
          Subbu M Iyer added a comment -

          So i think the consensus is to

          1) start with writing the descriptor to HDFS rather than ZK?
          2) Also, I believe that the data format should be in JSON in the file or ZK which ever it may be?

          Any preference on the JSON library (Jetty util/Jersey/Jettison/jackson) to use?

          Show
          Subbu M Iyer added a comment - So i think the consensus is to 1) start with writing the descriptor to HDFS rather than ZK? 2) Also, I believe that the data format should be in JSON in the file or ZK which ever it may be? Any preference on the JSON library (Jetty util/Jersey/Jettison/jackson) to use?
          Hide
          stack added a comment -

          Warning. This is a big one Subbu.

          So, I used to think that storing the schema in zk was the way to go but Ryan argues, correctly I believe, that zk should only carry transient data if only so we can copy hdfs content and then we can bring up the data elsewhere under another cluster (Otherwise, we'll have to copy hdfs and zk state to replicate cluster data elsewhere – a pain). So, we could write schema into a table or into hdfs. We could write the table schema into a new catalog table named schemas or, I believe it was Andrew Purtell who suggested, we put the schema into a new column family in .META. table into the first region only. If we wrote it into hdfs, we could write it into a .tabledescriptor file as we write the .regioninfo file now under each region. On startup, I'd think that we'd read hdfs or a schema table and then per table add a znode up in zk. On each new table edit, we'd update the znode. All regionservers would be watching the zk table proxy and would know to reread the schema on watcher trigger.

          On JSON serializing, yeah, that'd be sweet but might be a bit much to bite off as part of this issue. Maybe just go w/ Writables until its all running? Open new issue to add serialization of types to JSON (Todd mentions that if we avro'd this stuff, we could make use of an avro-to-json gateway that apparently avro has).

          Show
          stack added a comment - Warning. This is a big one Subbu. So, I used to think that storing the schema in zk was the way to go but Ryan argues, correctly I believe, that zk should only carry transient data if only so we can copy hdfs content and then we can bring up the data elsewhere under another cluster (Otherwise, we'll have to copy hdfs and zk state to replicate cluster data elsewhere – a pain). So, we could write schema into a table or into hdfs. We could write the table schema into a new catalog table named schemas or, I believe it was Andrew Purtell who suggested, we put the schema into a new column family in .META. table into the first region only. If we wrote it into hdfs, we could write it into a .tabledescriptor file as we write the .regioninfo file now under each region. On startup, I'd think that we'd read hdfs or a schema table and then per table add a znode up in zk. On each new table edit, we'd update the znode. All regionservers would be watching the zk table proxy and would know to reread the schema on watcher trigger. On JSON serializing, yeah, that'd be sweet but might be a bit much to bite off as part of this issue. Maybe just go w/ Writables until its all running? Open new issue to add serialization of types to JSON (Todd mentions that if we avro'd this stuff, we could make use of an avro-to-json gateway that apparently avro has).
          Hide
          Subbu M Iyer added a comment -

          Here is a brief summary of changes for this JIRA:

          1. HTD will be stored in HDFS in .tableinfo file under the table folder.
          2. HTD removed from HRI
          3. Bootstrap creates ROOT and META with out HTD in HRI. (for all fresh setup/startup. This process will
          also set a flag in ROOT called metamigrated and set it to true to indicate that the system supports
          new HRI's with out HTD.)
          4. HRegion.createHRegion stores HTD in HDFS. (Master also stores HTD in HDFS during table creation,
          but it seems redundant and will be removed).
          5. New HRegion instances will load the HTD of the table from HDFS.
          6. AssignmentManager maintains an in-memory map of <tableName, HTD> that gets modified during table
          manipulations (such as modifyTable, AddFamily, DeleteFamily etc) as well as during user region
          assignments.
          7. All schema operations such as modifyTable, AddFamily, DeleteFamily et al will result in appropriate
          changes to HTD on HDFS (.tableinfo changes).
          8. For existing installations:
          If the system is not updated (metamigrated is False or NULL in ROOT) to support HRI with out HTD, then
          HMaster at startup (before assigning ROOT and META) does 1-3 else jumps to 4.
          1. Creates new .tableInfo files for all tables in the system (based on ROOT and .META. scan)
          2. Updates existing META.HRI (with HTD) with the new version of HRI (with out HTD).
          3. Sets a flag in ROOT (metamigrated) to record this process and to avoid duplicate processing.
          4. Continue with regular startup.
          9. Appropriate changes to source and test files to work with new HRI structure with out HTD.

          Show
          Subbu M Iyer added a comment - Here is a brief summary of changes for this JIRA: 1. HTD will be stored in HDFS in .tableinfo file under the table folder. 2. HTD removed from HRI 3. Bootstrap creates ROOT and META with out HTD in HRI. (for all fresh setup/startup. This process will also set a flag in ROOT called metamigrated and set it to true to indicate that the system supports new HRI's with out HTD.) 4. HRegion.createHRegion stores HTD in HDFS. (Master also stores HTD in HDFS during table creation, but it seems redundant and will be removed). 5. New HRegion instances will load the HTD of the table from HDFS. 6. AssignmentManager maintains an in-memory map of <tableName, HTD> that gets modified during table manipulations (such as modifyTable, AddFamily, DeleteFamily etc) as well as during user region assignments. 7. All schema operations such as modifyTable, AddFamily, DeleteFamily et al will result in appropriate changes to HTD on HDFS (.tableinfo changes). 8. For existing installations: If the system is not updated (metamigrated is False or NULL in ROOT ) to support HRI with out HTD, then HMaster at startup (before assigning ROOT and META) does 1-3 else jumps to 4. 1. Creates new .tableInfo files for all tables in the system (based on ROOT and .META. scan) 2. Updates existing META.HRI (with HTD) with the new version of HRI (with out HTD). 3. Sets a flag in ROOT (metamigrated) to record this process and to avoid duplicate processing. 4. Continue with regular startup. 9. Appropriate changes to source and test files to work with new HRI structure with out HTD.
          Hide
          Ted Yu added a comment -

          This is the patch which applies to trunk.

          Subbu:
          https://reviews.apache.org/ should be used for this big change.

          Show
          Ted Yu added a comment - This is the patch which applies to trunk. Subbu: https://reviews.apache.org/ should be used for this big change.
          Hide
          Subbu M Iyer added a comment -

          Hi Ted,

          Could you please help me with posting a new review request?

          I am getting the following error when I try to upload my patch.
          "Unable to parse diff revision header '(revision
          a13cc6087b9b0d6924c32d100b612eedf01fd3c4)'"

          I am using Git for my version control. Do I need to do it through the RBTools?

          Please let me know.

          thanks
          Subbu

          Show
          Subbu M Iyer added a comment - Hi Ted, Could you please help me with posting a new review request? I am getting the following error when I try to upload my patch. "Unable to parse diff revision header '(revision a13cc6087b9b0d6924c32d100b612eedf01fd3c4)'" I am using Git for my version control. Do I need to do it through the RBTools? Please let me know. thanks Subbu
          Hide
          Ted Yu added a comment -

          I tried creating review myself with 451_support_for_removing_HTD_from_HRI_trunk.txt
          I got:
          The file '/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java' could not be found in the repository

          Show
          Ted Yu added a comment - I tried creating review myself with 451_support_for_removing_HTD_from_HRI_trunk.txt I got: The file '/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java' could not be found in the repository
          Hide
          Subbu M Iyer added a comment -

          Yes, because its a new addition. This patch has added two new files.

          1. '/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java
          2. /src/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java

          Please let me know how to address this.

          thanks
          Subbu

          Show
          Subbu M Iyer added a comment - Yes, because its a new addition. This patch has added two new files. 1. '/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java 2. /src/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java Please let me know how to address this. thanks Subbu
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/849/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          Posting for Subbu See issue for his description of change.

          This addresses bug HBASE-451.
          https://issues.apache.org/jira/browse/HBASE-451

          Diffs


          src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c
          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d
          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION
          src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800
          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0
          src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197
          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822
          src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0
          src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd
          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30
          src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b
          src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d
          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39
          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8
          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7
          src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac
          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4
          src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17
          src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20
          src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520
          src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47
          src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150
          src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483
          src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c
          src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154
          src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248
          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad
          src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6
          src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947
          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108
          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc
          src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803
          src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97
          src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788
          src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815
          src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703
          src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc
          src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509
          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe
          src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION
          src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb
          src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a
          src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3
          src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72
          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6
          src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0
          src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997
          src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767
          src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788
          src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507
          src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162
          src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e
          src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371
          src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2
          src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1
          src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f
          src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95
          src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a
          src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda
          src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055

          Diff: https://reviews.apache.org/r/849/diff

          Testing
          -------

          Thanks,

          Michael

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/ ----------------------------------------------------------- Review request for hbase. Summary ------- Posting for Subbu See issue for his description of change. This addresses bug HBASE-451 . https://issues.apache.org/jira/browse/HBASE-451 Diffs src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800 src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0 src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822 src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0 src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30 src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4 src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17 src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20 src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520 src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47 src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150 src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483 src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6 src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0 src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803 src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97 src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815 src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703 src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192 src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6 src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3 src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6 src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0 src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997 src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767 src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788 src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507 src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162 src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371 src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2 src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912 src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1 src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95 src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0 src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2 src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055 Diff: https://reviews.apache.org/r/849/diff Testing ------- Thanks, Michael
          Hide
          stack added a comment -

          I mucked around and got it posted by applying to my own git repo then made new 'plain' git patch. That uploaded to hbase.git on reviews.apache.org. My guess is that its dumb and explains output of git diff at top-level of hbase checkout.

          Show
          stack added a comment - I mucked around and got it posted by applying to my own git repo then made new 'plain' git patch. That uploaded to hbase.git on reviews.apache.org. My guess is that its dumb and explains output of git diff at top-level of hbase checkout.
          Hide
          Subbu M Iyer added a comment -

          Thanks Stack for your help. I wouldn't have gotten through this blocker myself.

          thanks again.

          Show
          Subbu M Iyer added a comment - Thanks Stack for your help. I wouldn't have gotten through this blocker myself. thanks again.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/849/#review749
          -----------------------------------------------------------

          This looks excellent. Lets get it committed before it rots. Do we still need .regioninfo after this goes in?

          src/main/java/org/apache/hadoop/hbase/HConstants.java
          <https://reviews.apache.org/r/849/#comment1541>

          Copy/paste error.

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1542>

          WhooHoo... this is starting out good!

          I think you should take the opportunity of changing HRI to be a VersionedWritable.

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1543>

          Excellent! (HRI not having ref to HTD)

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1544>

          This works? Thats interesting.

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1545>

          When would this be used?

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1546>

          This should not be allowed? Are you not going to mess up stuff if table gets changed on an HRI?

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1547>

          Should these be deprecated rather than removed? Even if we returned a null only?

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1548>

          This is perverse, begin able to change the HTD under an HRI. Can't believe this actually was allowed exist.

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1549>

          You could just do an equals since its boolean... Bytes.equals...

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1550>

          ditto

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <https://reviews.apache.org/r/849/#comment1551>

          This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member.

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java
          <https://reviews.apache.org/r/849/#comment1552>

          Put this into a migration subpackage?

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java
          <https://reviews.apache.org/r/849/#comment1554>

          Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI.

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java
          <https://reviews.apache.org/r/849/#comment1553>

          This is from elsewhere? Does it belong here?

          src/main/java/org/apache/hadoop/hbase/KeyValue.java
          <https://reviews.apache.org/r/849/#comment1555>

          I don't think we should do this by default. What if KV has a value of MBs?

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          <https://reviews.apache.org/r/849/#comment1557>

          What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style?

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
          <https://reviews.apache.org/r/849/#comment1556>

          You want to leave these in place?

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          <https://reviews.apache.org/r/849/#comment1558>

          What a crazy way of getting HTDs.

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
          <https://reviews.apache.org/r/849/#comment1559>

          Do we need these last two? Can't we use this first method to get the latter?

          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          <https://reviews.apache.org/r/849/#comment1560>

          Is this going to the FS to read tables? And we're holding sync while we are doing it? Is that necessary?

          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
          <https://reviews.apache.org/r/849/#comment1561>

          This method belongs in master since its going to run the migration?

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          <https://reviews.apache.org/r/849/#comment1562>

          Close the file?

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          <https://reviews.apache.org/r/849/#comment1563>

          Good. The human readable version is included.

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
          <https://reviews.apache.org/r/849/#comment1564>

          I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo?

          • Michael

          On 2011-06-02 20:52:15, Michael Stack wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/849/

          -----------------------------------------------------------

          (Updated 2011-06-02 20:52:15)

          Review request for hbase.

          Summary

          -------

          Posting for Subbu See issue for his description of change.

          This addresses bug HBASE-451.

          https://issues.apache.org/jira/browse/HBASE-451

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0

          src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822

          src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0

          src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30

          src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b

          src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39

          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4

          src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17

          src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20

          src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520

          src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47

          src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150

          src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483

          src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c

          src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154

          src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad

          src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6

          src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0

          src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc

          src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df

          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803

          src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97

          src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791

          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

          src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815

          src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703

          src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc

          src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509

          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe

          src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb

          src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a

          src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3

          src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6

          src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0

          src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997

          src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767

          src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788

          src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507

          src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162

          src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371

          src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912

          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1

          src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f

          src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95

          src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a

          src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda

          src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055

          Diff: https://reviews.apache.org/r/849/diff

          Testing

          -------

          Thanks,

          Michael

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/#review749 ----------------------------------------------------------- This looks excellent. Lets get it committed before it rots. Do we still need .regioninfo after this goes in? src/main/java/org/apache/hadoop/hbase/HConstants.java < https://reviews.apache.org/r/849/#comment1541 > Copy/paste error. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1542 > WhooHoo... this is starting out good! I think you should take the opportunity of changing HRI to be a VersionedWritable. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1543 > Excellent! (HRI not having ref to HTD) src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1544 > This works? Thats interesting. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1545 > When would this be used? src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1546 > This should not be allowed? Are you not going to mess up stuff if table gets changed on an HRI? src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1547 > Should these be deprecated rather than removed? Even if we returned a null only? src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1548 > This is perverse, begin able to change the HTD under an HRI. Can't believe this actually was allowed exist. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1549 > You could just do an equals since its boolean... Bytes.equals... src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1550 > ditto src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < https://reviews.apache.org/r/849/#comment1551 > This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member. src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java < https://reviews.apache.org/r/849/#comment1552 > Put this into a migration subpackage? src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java < https://reviews.apache.org/r/849/#comment1554 > Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI. src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java < https://reviews.apache.org/r/849/#comment1553 > This is from elsewhere? Does it belong here? src/main/java/org/apache/hadoop/hbase/KeyValue.java < https://reviews.apache.org/r/849/#comment1555 > I don't think we should do this by default. What if KV has a value of MBs? src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java < https://reviews.apache.org/r/849/#comment1557 > What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style? src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java < https://reviews.apache.org/r/849/#comment1556 > You want to leave these in place? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java < https://reviews.apache.org/r/849/#comment1558 > What a crazy way of getting HTDs. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java < https://reviews.apache.org/r/849/#comment1559 > Do we need these last two? Can't we use this first method to get the latter? src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java < https://reviews.apache.org/r/849/#comment1560 > Is this going to the FS to read tables? And we're holding sync while we are doing it? Is that necessary? src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java < https://reviews.apache.org/r/849/#comment1561 > This method belongs in master since its going to run the migration? src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java < https://reviews.apache.org/r/849/#comment1562 > Close the file? src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java < https://reviews.apache.org/r/849/#comment1563 > Good. The human readable version is included. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java < https://reviews.apache.org/r/849/#comment1564 > I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo? Michael On 2011-06-02 20:52:15, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/ ----------------------------------------------------------- (Updated 2011-06-02 20:52:15) Review request for hbase. Summary ------- Posting for Subbu See issue for his description of change. This addresses bug HBASE-451 . https://issues.apache.org/jira/browse/HBASE-451 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800 src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0 src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822 src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0 src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30 src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4 src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17 src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20 src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520 src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47 src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150 src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483 src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6 src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0 src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803 src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97 src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815 src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703 src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192 src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6 src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3 src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6 src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0 src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997 src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767 src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788 src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507 src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162 src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371 src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2 src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912 src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1 src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95 src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0 src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2 src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055 Diff: https://reviews.apache.org/r/849/diff Testing ------- Thanks, Michael
          Hide
          Ted Yu added a comment -

          I ran test suite for the patch.
          There were a lot of failures.
          Here is one sample:

          rg.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion  Time elapsed: 0 sec  <<< ERROR!
          org.apache.hadoop.hbase.TableNotFoundException: TestZKBasedOpenCloseRegion
                  at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1752)
                  at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:348)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:818)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:786)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:760)
                  at org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion.beforeAllTests(TestZKBasedOpenCloseRegion.java:72)
          
          Show
          Ted Yu added a comment - I ran test suite for the patch. There were a lot of failures. Here is one sample: rg.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion Time elapsed: 0 sec <<< ERROR! org.apache.hadoop.hbase.TableNotFoundException: TestZKBasedOpenCloseRegion at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1752) at org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:348) at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:818) at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:786) at org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:760) at org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion.beforeAllTests(TestZKBasedOpenCloseRegion.java:72)
          Hide
          Subbu M Iyer added a comment -

          Hi Ted,

          Let me take a look at it. For reference I have attached my test results as well. All tests were passing in my local.

          ==================
          /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBase451/hbase/pom.xml test
          [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions.
          [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions.
          [INFO] Scanning for projects...
          [WARNING]
          [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT
          [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 531, column 15
          [WARNING]
          [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
          [WARNING]
          [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
          [WARNING]
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building HBase 0.91.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml
          680 B

          Downloaded: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml (680 B at 1.4 KB/sec)
          [INFO]
          [INFO] — maven-antrun-plugin:1.3:run (generate) @ hbase —
          [INFO] Executing tasks
          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          2011-05-31 18:57:10.472:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
          [exec] /work/HBase451/hbase/src/saveVersion.sh: line 32: git: command not found
          [INFO] Executed tasks
          [INFO]
          [INFO] — build-helper-maven-plugin:1.5:add-source (add-jspc-source) @ hbase —
          [INFO] Source directory: /work/HBase451/hbase/target/jspc added.
          [INFO]
          [INFO] — build-helper-maven-plugin:1.5:add-source (add-package-info) @ hbase —
          [INFO] Source directory: /work/HBase451/hbase/target/generated-sources added.
          [INFO]
          [INFO] — jamon-maven-plugin:2.3.4:translate (default) @ hbase —
          [INFO]
          [INFO] — maven-remote-resources-plugin:1.1:process (default) @ hbase —
          [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
          [INFO] Setting property: velocimacro.messages.on => 'false'.
          [INFO] Setting property: resource.loader => 'classpath'.
          [INFO] Setting property: resource.manager.logwhenfound => 'false'.
          [INFO]
          [INFO] — maven-resources-plugin:2.4:resources (default-resources) @ hbase —
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 1 resource
          [INFO] Copying 6 resources
          [INFO] Copying 3 resources
          [INFO]
          [INFO] — maven-antrun-plugin:1.3:run (default) @ hbase —
          [INFO] Executing tasks
          [INFO] Executed tasks
          [INFO]
          [INFO] — maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase —
          [INFO] Compiling 8 source files to /work/HBase451/hbase/target/classes
          [INFO]
          [INFO] — maven-resources-plugin:2.4:testResources (default-testResources) @ hbase —
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 4 resources
          [INFO] Copying 3 resources
          [INFO]
          [INFO] — maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase —
          [INFO] Nothing to compile - all classes are up to date
          [INFO]
          [INFO] — maven-surefire-plugin:2.4.3:test (default-test) @ hbase —
          [INFO] Surefire report directory: /work/HBase451/hbase/target/surefire-reports
          2011-05-31 18:57:18,255 INFO [main] hbase.HRegionInfo(260): ******* New HRegionInfo testscanner region id = 1306893438255

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hbase.master.TestHMasterRPCException
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.726 sec
          Running org.apache.hadoop.hbase.regionserver.TestColumnSeeking
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.908 sec
          Running org.apache.hadoop.hbase.client.TestMultipleTimestamps
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.488 sec
          Running org.apache.hadoop.hbase.regionserver.TestMemStoreLAB
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec
          Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.715 sec
          Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.508 sec
          Running org.apache.hadoop.hbase.TestZooKeeper
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.267 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.148 sec
          Running org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.159 sec
          Running org.apache.hadoop.hbase.filter.TestPrefixFilter
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.069 sec
          Running org.apache.hadoop.hbase.io.TestHeapSize
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084 sec
          Running org.apache.hadoop.hbase.io.TestImmutableBytesWritable
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec
          Running org.apache.hadoop.hbase.rest.model.TestRowModel
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203 sec
          Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.886 sec
          Running org.apache.hadoop.hbase.regionserver.TestStore
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.659 sec
          Running org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.995 sec
          Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.397 sec
          Running org.apache.hadoop.hbase.rest.TestScannersWithFilters
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.018 sec
          Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.311 sec
          Running org.apache.hadoop.hbase.io.TestHbaseObjectWritable
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.139 sec
          Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.781 sec
          Running org.apache.hadoop.hbase.regionserver.TestResettingCounters
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.627 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
          Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 202.441 sec
          Running org.apache.hadoop.hbase.thrift.TestThriftServer
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.041 sec
          Running org.apache.hadoop.hbase.TestHServerInfo
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1 sec
          Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
          Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 130.844 sec <<< FAILURE!
          Running org.apache.hadoop.hbase.filter.TestColumnPrefixFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.959 sec
          Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.438 sec
          Running org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.073 sec
          Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.396 sec
          Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec
          Running org.apache.hadoop.hbase.TestCompare
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.077 sec
          Running org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.54 sec
          Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 sec
          Running org.apache.hadoop.hbase.io.TestHalfStoreFileReader
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec
          Running org.apache.hadoop.hbase.rest.TestTransform
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.794 sec
          Running org.apache.hadoop.hbase.util.TestFSUtils
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.239 sec
          Running org.apache.hadoop.hbase.filter.TestBitComparator
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec
          Running org.apache.hadoop.hbase.security.TestUser
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.138 sec
          Running org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.225 sec
          Running org.apache.hadoop.hbase.master.TestMasterTransitions
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 21.518 sec
          Running org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec
          Running org.apache.hadoop.hbase.io.hfile.TestSeekTo
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.389 sec
          Running org.apache.hadoop.hbase.filter.TestPageFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028 sec
          Running org.apache.hadoop.hbase.filter.TestColumnPaginationFilter
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec
          Running org.apache.hadoop.hbase.rest.TestStatusResource
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.806 sec
          Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.339 sec
          Running org.apache.hadoop.hbase.executor.TestExecutorService
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.226 sec
          Running org.apache.hadoop.hbase.client.TestFromClientSide
          Tests run: 44, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 125.272 sec
          Running org.apache.hadoop.hbase.replication.TestReplication
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.036 sec
          Running org.apache.hadoop.hbase.regionserver.TestCompaction
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.772 sec
          Running org.apache.hadoop.hbase.TestHServerAddress
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec
          Running org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec
          Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.995 sec
          Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.056 sec
          Running org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec
          Running org.apache.hadoop.hbase.master.TestClockSkewDetection
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.131 sec
          Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 138.375 sec
          Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.452 sec
          Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.534 sec
          Running org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.207 sec
          Running org.apache.hadoop.hbase.client.TestMetaScanner
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.628 sec
          Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec
          Running org.apache.hadoop.hbase.TestRegionRebalancing
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 331.575 sec
          Running org.apache.hadoop.hbase.rest.TestMultiRowResource
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.495 sec
          Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.282 sec
          Running org.apache.hadoop.hbase.rest.TestTableResource
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.208 sec
          Running org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 sec
          Running org.apache.hadoop.hbase.filter.TestInclusiveStopFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.106 sec
          Running org.apache.hadoop.hbase.TestHBaseTestingUtility
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.578 sec
          Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.332 sec
          Running org.apache.hadoop.hbase.rest.TestSchemaResource
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.673 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.975 sec
          Running org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 sec
          Running org.apache.hadoop.hbase.rest.model.TestTableListModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.26 sec
          Running org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.076 sec
          Running org.apache.hadoop.hbase.regionserver.TestStoreFile
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.452 sec
          Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.507 sec
          Running org.apache.hadoop.hbase.rest.model.TestScannerModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.251 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.831 sec
          Running org.apache.hadoop.hbase.master.TestSplitLogManager
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.479 sec
          Running org.apache.hadoop.hbase.rest.TestVersionResource
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.053 sec
          Running org.apache.hadoop.hbase.filter.TestFilter
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.011 sec
          Running org.apache.hadoop.hbase.TestServerName
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 sec
          Running org.apache.hadoop.hbase.regionserver.TestWideScanner
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.406 sec
          Running org.apache.hadoop.hbase.regionserver.TestSplitTransaction
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.274 sec
          Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.718 sec
          Running org.apache.hadoop.hbase.util.TestMergeTable
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.097 sec
          Running org.apache.hadoop.hbase.regionserver.TestGetClosestAtOrBefore
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.171 sec
          Running org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.116 sec
          Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.696 sec
          Running org.apache.hadoop.hbase.rest.model.TestVersionModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.217 sec
          Running org.apache.hadoop.hbase.rest.model.TestCellSetModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.264 sec
          Running org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.215 sec
          Running org.apache.hadoop.hbase.rest.client.TestRemoteTable
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.118 sec
          Running org.apache.hadoop.hbase.monitoring.TestTaskMonitor
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec
          Running org.apache.hadoop.hbase.avro.TestAvroServer
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.918 sec
          Running org.apache.hadoop.hbase.TestFullLogReconstruction
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.043 sec
          Running org.apache.hadoop.hbase.TestSerialization
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.131 sec
          Running org.apache.hadoop.hbase.regionserver.TestKeyValueHeap
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec
          Running org.apache.hadoop.hbase.regionserver.TestStoreScanner
          Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec
          Running org.apache.hadoop.hbase.io.hfile.TestReseekTo
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec
          Running org.apache.hadoop.hbase.util.TestKeying
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 sec
          Running org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec
          Running org.apache.hadoop.hbase.regionserver.TestScanner
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.096 sec
          Running org.apache.hadoop.hbase.regionserver.TestHRegionInfo
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.087 sec
          Running org.apache.hadoop.hbase.util.TestHBaseFsck
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.581 sec
          Running org.apache.hadoop.hbase.client.TestShell
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.664 sec
          Running org.apache.hadoop.hbase.master.TestRestartCluster
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.963 sec
          Running org.apache.hadoop.hbase.rest.model.TestTableInfoModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 sec
          Running org.apache.hadoop.hbase.TestInfoServers
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.196 sec
          Running org.apache.hadoop.hbase.master.TestLogsCleaner
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.877 sec
          Running org.apache.hadoop.hbase.regionserver.TestMasterAddressManager
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.443 sec
          Running org.apache.hadoop.hbase.master.TestMasterStatusServlet
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.899 sec
          Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.701 sec
          Running org.apache.hadoop.hbase.regionserver.TestQueryMatcher
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.262 sec
          Running org.apache.hadoop.hbase.zookeeper.TestZKTable
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.435 sec
          Running org.apache.hadoop.hbase.client.TestAdmin
          Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 372.551 sec <<< FAILURE!
          Running org.apache.hadoop.hbase.util.TestSortedCopyOnWriteSet
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec
          Running org.apache.hadoop.hbase.util.TestMergeTool
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.054 sec
          Running org.apache.hadoop.hbase.TestAcidGuarantees
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.044 sec
          Running org.apache.hadoop.hbase.TestHRegionLocation
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec
          Running org.apache.hadoop.hbase.util.TestBase64
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec
          Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.605 sec
          Running org.apache.hadoop.hbase.regionserver.TestMemStore
          Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.688 sec
          Running org.apache.hadoop.hbase.rest.TestScannerResource
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.001 sec
          Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 sec
          Running org.apache.hadoop.hbase.master.TestDeadServer
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec
          Running org.apache.hadoop.hbase.rest.TestGzipFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.005 sec
          Running org.apache.hadoop.hbase.TestKeyValue
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec
          Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.569 sec
          Running org.apache.hadoop.hbase.filter.TestColumnRangeFilter
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.707 sec
          Running org.apache.hadoop.hbase.client.TestMultiParallel
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.108 sec
          Running org.apache.hadoop.hbase.master.TestMaster
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.511 sec
          Running org.apache.hadoop.hbase.client.TestResult
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec
          Running org.apache.hadoop.hbase.master.TestRollingRestart
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.231 sec
          Running org.apache.hadoop.hbase.client.TestHTablePool
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 158.175 sec
          Running org.apache.hadoop.hbase.regionserver.TestRSStatusServlet
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.454 sec
          Running org.apache.hadoop.hbase.util.TestByteBloomFilter
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.532 sec
          Running org.apache.hadoop.hbase.rest.model.TestTableSchemaModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.21 sec
          Running org.apache.hadoop.hbase.regionserver.TestCompactSelection
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.553 sec
          Running org.apache.hadoop.hbase.filter.TestFilterList
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec
          Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 586.446 sec
          Running org.apache.hadoop.hbase.regionserver.TestHRegion
          Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.679 sec
          Running org.apache.hadoop.hbase.rest.model.TestTableRegionModel
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec
          Running org.apache.hadoop.hbase.master.TestMasterFailover
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.064 sec
          Running org.apache.hadoop.hbase.client.TestTimestampsFilter
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.717 sec
          Running org.apache.hadoop.hbase.util.TestRootPath
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.035 sec
          Running org.apache.hadoop.hbase.master.TestLoadBalancer
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.182 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestWALObserver
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 sec
          Running org.apache.hadoop.hbase.client.TestScannerTimeout
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.297 sec
          Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.89 sec
          Running org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
          Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.331 sec
          Running org.apache.hadoop.hbase.util.TestBytes
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec
          Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.093 sec
          Running org.apache.hadoop.hbase.filter.TestRandomRowFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.446 sec
          Running org.apache.hadoop.hbase.master.TestActiveMasterManager
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.6 sec
          Running org.apache.hadoop.hbase.util.TestCompressionTest
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.248 sec
          Running org.apache.hadoop.hbase.client.TestMetaMigration
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.579 sec
          Running org.apache.hadoop.hbase.filter.TestDependentColumnFilter
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.782 sec
          Running org.apache.hadoop.hbase.client.TestGetRowVersions
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.784 sec
          Running org.apache.hadoop.hbase.rest.model.TestCellModel
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.245 sec
          Running org.apache.hadoop.hbase.replication.TestReplicationSource
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.156 sec
          Running org.apache.hadoop.hbase.coprocessor.TestWALObserver
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.317 sec
          Running org.apache.hadoop.hbase.catalog.TestCatalogTracker
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.259 sec
          Running org.apache.hadoop.hbase.master.TestCatalogJanitor
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.113 sec
          Running org.apache.hadoop.hbase.rest.TestRowResource
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.772 sec
          Running org.apache.hadoop.hbase.io.hfile.TestHFile
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.646 sec
          Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.576 sec
          Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.975 sec
          Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.872 sec
          Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.434 sec
          Running org.apache.hadoop.hbase.client.TestTimestamp
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.394 sec
          Running org.apache.hadoop.hbase.util.TestPoolMap
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111 sec
          Running org.apache.hadoop.hbase.TestScanMultipleVersions
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.018 sec
          Running org.apache.hadoop.hbase.TestGlobalMemStoreSize
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.676 sec
          Running org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 sec
          Running org.apache.hadoop.hbase.client.TestHCM
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.962 sec

          Results :

          Failed tests:

          Tests in error:

          Tests run: 795, Failures: 1, Errors: 3, Skipped: 9

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:52:50.048s
          [INFO] Finished at: Tue May 31 20:49:55 PDT 2011
          [INFO] Final Memory: 17M/81M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test (default-test) on project hbase: There are test failures.
          [ERROR]
          [ERROR] Please refer to /work/HBase451/hbase/target/surefire-reports for the individual test results.
          [ERROR] -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

          Process finished with exit code 1
          =======================================

          subbumac:hbase subbu$ mvn -Dtest=TestDistributedLogSplitting test
          [INFO] Scanning for projects...
          [WARNING]
          [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT
          [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 531, column 15
          [WARNING]
          [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
          [WARNING]
          [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
          [WARNING]
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building HBase 0.91.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO]
          [INFO] — maven-antrun-plugin:1.3:run (generate) @ hbase —
          [INFO] Executing tasks
          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          2011-05-31 19:47:06.018:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
          [INFO] Executed tasks
          [INFO]
          [INFO] — build-helper-maven-plugin:1.5:add-source (add-jspc-source) @ hbase —
          [INFO] Source directory: /work/HBase451/hbase/target/jspc added.
          [INFO]
          [INFO] — build-helper-maven-plugin:1.5:add-source (add-package-info) @ hbase —
          [INFO] Source directory: /work/HBase451/hbase/target/generated-sources added.
          [INFO]
          [INFO] — jamon-maven-plugin:2.3.4:translate (default) @ hbase —
          [INFO]
          [INFO] — maven-remote-resources-plugin:1.1:process (default) @ hbase —
          [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'.
          [INFO] Setting property: velocimacro.messages.on => 'false'.
          [INFO] Setting property: resource.loader => 'classpath'.
          [INFO] Setting property: resource.manager.logwhenfound => 'false'.
          [INFO]
          [INFO] — maven-resources-plugin:2.4:resources (default-resources) @ hbase —
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 1 resource
          [INFO] Copying 6 resources
          [INFO] Copying 3 resources
          [INFO]
          [INFO] — maven-antrun-plugin:1.3:run (default) @ hbase —
          [INFO] Executing tasks
          [INFO] Executed tasks
          [INFO]
          [INFO] — maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase —
          [INFO] Compiling 8 source files to /work/HBase451/hbase/target/classes
          [INFO]
          [INFO] — maven-resources-plugin:2.4:testResources (default-testResources) @ hbase —
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 4 resources
          [INFO] Copying 3 resources
          [INFO]
          [INFO] — maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase —
          [INFO] Compiling 1 source file to /work/HBase451/hbase/target/test-classes
          [INFO]
          [INFO] — maven-surefire-plugin:2.4.3:test (default-test) @ hbase —
          [INFO] Surefire report directory: /work/HBase451/hbase/target/surefire-reports

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.136 sec

          Results :

          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:36.596s
          [INFO] Finished at: Tue May 31 19:49:35 PDT 2011
          [INFO] Final Memory: 19M/81M
          [INFO] ------------------------------------------------------------------------

          =================================================================================================

          Show
          Subbu M Iyer added a comment - Hi Ted, Let me take a look at it. For reference I have attached my test results as well. All tests were passing in my local. ================== /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBase451/hbase/pom.xml test [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions. [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions. [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 531, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase 0.91.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml 680 B Downloaded: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml (680 B at 1.4 KB/sec) [INFO] [INFO] — maven-antrun-plugin:1.3:run (generate) @ hbase — [INFO] Executing tasks SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2011-05-31 18:57:10.472:INFO::Logging to STDERR via org.mortbay.log.StdErrLog [exec] /work/HBase451/hbase/src/saveVersion.sh: line 32: git: command not found [INFO] Executed tasks [INFO] [INFO] — build-helper-maven-plugin:1.5:add-source (add-jspc-source) @ hbase — [INFO] Source directory: /work/HBase451/hbase/target/jspc added. [INFO] [INFO] — build-helper-maven-plugin:1.5:add-source (add-package-info) @ hbase — [INFO] Source directory: /work/HBase451/hbase/target/generated-sources added. [INFO] [INFO] — jamon-maven-plugin:2.3.4:translate (default) @ hbase — [INFO] [INFO] — maven-remote-resources-plugin:1.1:process (default) @ hbase — [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on => 'false'. [INFO] Setting property: resource.loader => 'classpath'. [INFO] Setting property: resource.manager.logwhenfound => 'false'. [INFO] [INFO] — maven-resources-plugin:2.4:resources (default-resources) @ hbase — [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 6 resources [INFO] Copying 3 resources [INFO] [INFO] — maven-antrun-plugin:1.3:run (default) @ hbase — [INFO] Executing tasks [INFO] Executed tasks [INFO] [INFO] — maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase — [INFO] Compiling 8 source files to /work/HBase451/hbase/target/classes [INFO] [INFO] — maven-resources-plugin:2.4:testResources (default-testResources) @ hbase — [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 3 resources [INFO] [INFO] — maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase — [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] — maven-surefire-plugin:2.4.3:test (default-test) @ hbase — [INFO] Surefire report directory: /work/HBase451/hbase/target/surefire-reports 2011-05-31 18:57:18,255 INFO [main] hbase.HRegionInfo(260): ******* New HRegionInfo testscanner region id = 1306893438255 ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.master.TestHMasterRPCException Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.726 sec Running org.apache.hadoop.hbase.regionserver.TestColumnSeeking Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.908 sec Running org.apache.hadoop.hbase.client.TestMultipleTimestamps Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.488 sec Running org.apache.hadoop.hbase.regionserver.TestMemStoreLAB Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.715 sec Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.508 sec Running org.apache.hadoop.hbase.TestZooKeeper Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.267 sec Running org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.148 sec Running org.apache.hadoop.hbase.io.hfile.TestCachedBlockQueue Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.159 sec Running org.apache.hadoop.hbase.filter.TestPrefixFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.069 sec Running org.apache.hadoop.hbase.io.TestHeapSize Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084 sec Running org.apache.hadoop.hbase.io.TestImmutableBytesWritable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec Running org.apache.hadoop.hbase.rest.model.TestRowModel Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203 sec Running org.apache.hadoop.hbase.io.hfile.TestHFileSeek Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.886 sec Running org.apache.hadoop.hbase.regionserver.TestStore Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.659 sec Running org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.995 sec Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.397 sec Running org.apache.hadoop.hbase.rest.TestScannersWithFilters Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.018 sec Running org.apache.hadoop.hbase.io.hfile.TestHFilePerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.311 sec Running org.apache.hadoop.hbase.io.TestHbaseObjectWritable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.139 sec Running org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.781 sec Running org.apache.hadoop.hbase.regionserver.TestResettingCounters Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.627 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 202.441 sec Running org.apache.hadoop.hbase.thrift.TestThriftServer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.041 sec Running org.apache.hadoop.hbase.TestHServerInfo Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.1 sec Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 130.844 sec <<< FAILURE! Running org.apache.hadoop.hbase.filter.TestColumnPrefixFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.959 sec Running org.apache.hadoop.hbase.regionserver.TestRpcMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.438 sec Running org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.073 sec Running org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.396 sec Running org.apache.hadoop.hbase.util.TestEnvironmentEdgeManager Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec Running org.apache.hadoop.hbase.TestCompare Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.077 sec Running org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.54 sec Running org.apache.hadoop.hbase.util.TestDefaultEnvironmentEdge Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 sec Running org.apache.hadoop.hbase.io.TestHalfStoreFileReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec Running org.apache.hadoop.hbase.rest.TestTransform Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.794 sec Running org.apache.hadoop.hbase.util.TestFSUtils Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.239 sec Running org.apache.hadoop.hbase.filter.TestBitComparator Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec Running org.apache.hadoop.hbase.security.TestUser Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.138 sec Running org.apache.hadoop.hbase.regionserver.TestScanWildcardColumnTracker Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.225 sec Running org.apache.hadoop.hbase.master.TestMasterTransitions Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 21.518 sec Running org.apache.hadoop.hbase.regionserver.TestKeyValueSkipListSet Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec Running org.apache.hadoop.hbase.io.hfile.TestSeekTo Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.389 sec Running org.apache.hadoop.hbase.filter.TestPageFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028 sec Running org.apache.hadoop.hbase.filter.TestColumnPaginationFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.027 sec Running org.apache.hadoop.hbase.rest.TestStatusResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.806 sec Running org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.339 sec Running org.apache.hadoop.hbase.executor.TestExecutorService Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.226 sec Running org.apache.hadoop.hbase.client.TestFromClientSide Tests run: 44, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 125.272 sec Running org.apache.hadoop.hbase.replication.TestReplication Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.036 sec Running org.apache.hadoop.hbase.regionserver.TestCompaction Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.772 sec Running org.apache.hadoop.hbase.TestHServerAddress Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec Running org.apache.hadoop.hbase.filter.TestSingleColumnValueFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.995 sec Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.056 sec Running org.apache.hadoop.hbase.mapreduce.TestSimpleTotalOrderPartitioner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec Running org.apache.hadoop.hbase.master.TestClockSkewDetection Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.131 sec Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 138.375 sec Running org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.452 sec Running org.apache.hadoop.hbase.client.replication.TestReplicationAdmin Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.534 sec Running org.apache.hadoop.hbase.regionserver.TestScanDeleteTracker Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.207 sec Running org.apache.hadoop.hbase.client.TestMetaScanner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.628 sec Running org.apache.hadoop.hbase.metrics.TestMetricsMBeanBase Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec Running org.apache.hadoop.hbase.TestRegionRebalancing Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 331.575 sec Running org.apache.hadoop.hbase.rest.TestMultiRowResource Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.495 sec Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.282 sec Running org.apache.hadoop.hbase.rest.TestTableResource Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.208 sec Running org.apache.hadoop.hbase.filter.TestSingleColumnValueExcludeFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 sec Running org.apache.hadoop.hbase.filter.TestInclusiveStopFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.106 sec Running org.apache.hadoop.hbase.TestHBaseTestingUtility Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.578 sec Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.332 sec Running org.apache.hadoop.hbase.rest.TestSchemaResource Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.673 sec Running org.apache.hadoop.hbase.regionserver.wal.TestWALReplay Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.975 sec Running org.apache.hadoop.hbase.regionserver.TestKeyValueScanFixture Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.109 sec Running org.apache.hadoop.hbase.rest.model.TestTableListModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.26 sec Running org.apache.hadoop.hbase.rest.model.TestColumnSchemaModel Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.076 sec Running org.apache.hadoop.hbase.regionserver.TestStoreFile Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.452 sec Running org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.507 sec Running org.apache.hadoop.hbase.rest.model.TestScannerModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.251 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.831 sec Running org.apache.hadoop.hbase.master.TestSplitLogManager Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.479 sec Running org.apache.hadoop.hbase.rest.TestVersionResource Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.053 sec Running org.apache.hadoop.hbase.filter.TestFilter Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.011 sec Running org.apache.hadoop.hbase.TestServerName Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.055 sec Running org.apache.hadoop.hbase.regionserver.TestWideScanner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.406 sec Running org.apache.hadoop.hbase.regionserver.TestSplitTransaction Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.274 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSink Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.718 sec Running org.apache.hadoop.hbase.util.TestMergeTable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.097 sec Running org.apache.hadoop.hbase.regionserver.TestGetClosestAtOrBefore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.171 sec Running org.apache.hadoop.hbase.regionserver.TestReadWriteConsistencyControl Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.116 sec Running org.apache.hadoop.hbase.coprocessor.TestMasterObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.696 sec Running org.apache.hadoop.hbase.rest.model.TestVersionModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.217 sec Running org.apache.hadoop.hbase.rest.model.TestCellSetModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.264 sec Running org.apache.hadoop.hbase.rest.model.TestStorageClusterStatusModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.215 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteTable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.118 sec Running org.apache.hadoop.hbase.monitoring.TestTaskMonitor Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.407 sec Running org.apache.hadoop.hbase.avro.TestAvroServer Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.918 sec Running org.apache.hadoop.hbase.TestFullLogReconstruction Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 63.043 sec Running org.apache.hadoop.hbase.TestSerialization Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.131 sec Running org.apache.hadoop.hbase.regionserver.TestKeyValueHeap Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.249 sec Running org.apache.hadoop.hbase.regionserver.TestStoreScanner Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec Running org.apache.hadoop.hbase.io.hfile.TestReseekTo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec Running org.apache.hadoop.hbase.util.TestKeying Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 sec Running org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec Running org.apache.hadoop.hbase.regionserver.TestScanner Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.096 sec Running org.apache.hadoop.hbase.regionserver.TestHRegionInfo Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.087 sec Running org.apache.hadoop.hbase.util.TestHBaseFsck Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.581 sec Running org.apache.hadoop.hbase.client.TestShell Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.664 sec Running org.apache.hadoop.hbase.master.TestRestartCluster Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.963 sec Running org.apache.hadoop.hbase.rest.model.TestTableInfoModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 sec Running org.apache.hadoop.hbase.TestInfoServers Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.196 sec Running org.apache.hadoop.hbase.master.TestLogsCleaner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.877 sec Running org.apache.hadoop.hbase.regionserver.TestMasterAddressManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.443 sec Running org.apache.hadoop.hbase.master.TestMasterStatusServlet Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.899 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 211.701 sec Running org.apache.hadoop.hbase.regionserver.TestQueryMatcher Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.262 sec Running org.apache.hadoop.hbase.zookeeper.TestZKTable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.435 sec Running org.apache.hadoop.hbase.client.TestAdmin Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 372.551 sec <<< FAILURE! Running org.apache.hadoop.hbase.util.TestSortedCopyOnWriteSet Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.054 sec Running org.apache.hadoop.hbase.TestAcidGuarantees Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.044 sec Running org.apache.hadoop.hbase.TestHRegionLocation Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec Running org.apache.hadoop.hbase.util.TestBase64 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.605 sec Running org.apache.hadoop.hbase.regionserver.TestMemStore Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.688 sec Running org.apache.hadoop.hbase.rest.TestScannerResource Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.001 sec Running org.apache.hadoop.hbase.io.hfile.TestLruBlockCache Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 sec Running org.apache.hadoop.hbase.master.TestDeadServer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec Running org.apache.hadoop.hbase.rest.TestGzipFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.005 sec Running org.apache.hadoop.hbase.TestKeyValue Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.569 sec Running org.apache.hadoop.hbase.filter.TestColumnRangeFilter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.707 sec Running org.apache.hadoop.hbase.client.TestMultiParallel Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.108 sec Running org.apache.hadoop.hbase.master.TestMaster Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.511 sec Running org.apache.hadoop.hbase.client.TestResult Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 sec Running org.apache.hadoop.hbase.master.TestRollingRestart Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 60.231 sec Running org.apache.hadoop.hbase.client.TestHTablePool Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 158.175 sec Running org.apache.hadoop.hbase.regionserver.TestRSStatusServlet Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.454 sec Running org.apache.hadoop.hbase.util.TestByteBloomFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.532 sec Running org.apache.hadoop.hbase.rest.model.TestTableSchemaModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.21 sec Running org.apache.hadoop.hbase.regionserver.TestCompactSelection Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.553 sec Running org.apache.hadoop.hbase.filter.TestFilterList Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.134 sec Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 586.446 sec Running org.apache.hadoop.hbase.regionserver.TestHRegion Tests run: 54, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.679 sec Running org.apache.hadoop.hbase.rest.model.TestTableRegionModel Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec Running org.apache.hadoop.hbase.master.TestMasterFailover Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 85.064 sec Running org.apache.hadoop.hbase.client.TestTimestampsFilter Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.717 sec Running org.apache.hadoop.hbase.util.TestRootPath Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.035 sec Running org.apache.hadoop.hbase.master.TestLoadBalancer Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.182 sec Running org.apache.hadoop.hbase.regionserver.wal.TestWALObserver Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.712 sec Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.297 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.89 sec Running org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.331 sec Running org.apache.hadoop.hbase.util.TestBytes Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec Running org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.093 sec Running org.apache.hadoop.hbase.filter.TestRandomRowFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.446 sec Running org.apache.hadoop.hbase.master.TestActiveMasterManager Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.6 sec Running org.apache.hadoop.hbase.util.TestCompressionTest Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.248 sec Running org.apache.hadoop.hbase.client.TestMetaMigration Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.579 sec Running org.apache.hadoop.hbase.filter.TestDependentColumnFilter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.782 sec Running org.apache.hadoop.hbase.client.TestGetRowVersions Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.784 sec Running org.apache.hadoop.hbase.rest.model.TestCellModel Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.245 sec Running org.apache.hadoop.hbase.replication.TestReplicationSource Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.156 sec Running org.apache.hadoop.hbase.coprocessor.TestWALObserver Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.317 sec Running org.apache.hadoop.hbase.catalog.TestCatalogTracker Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.259 sec Running org.apache.hadoop.hbase.master.TestCatalogJanitor Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.113 sec Running org.apache.hadoop.hbase.rest.TestRowResource Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.772 sec Running org.apache.hadoop.hbase.io.hfile.TestHFile Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.646 sec Running org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.576 sec Running org.apache.hadoop.hbase.mapreduce.TestImportTsv Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.975 sec Running org.apache.hadoop.hbase.rest.client.TestRemoteAdmin Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.872 sec Running org.apache.hadoop.hbase.regionserver.wal.TestHLog Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.434 sec Running org.apache.hadoop.hbase.client.TestTimestamp Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.394 sec Running org.apache.hadoop.hbase.util.TestPoolMap Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111 sec Running org.apache.hadoop.hbase.TestScanMultipleVersions Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.018 sec Running org.apache.hadoop.hbase.TestGlobalMemStoreSize Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.676 sec Running org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 sec Running org.apache.hadoop.hbase.client.TestHCM Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.962 sec Results : Failed tests: Tests in error: Tests run: 795, Failures: 1, Errors: 3, Skipped: 9 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:52:50.048s [INFO] Finished at: Tue May 31 20:49:55 PDT 2011 [INFO] Final Memory: 17M/81M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.4.3:test (default-test) on project hbase: There are test failures. [ERROR] [ERROR] Please refer to /work/HBase451/hbase/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException Process finished with exit code 1 ======================================= subbumac:hbase subbu$ mvn -Dtest=TestDistributedLogSplitting test [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 531, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase 0.91.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] — maven-antrun-plugin:1.3:run (generate) @ hbase — [INFO] Executing tasks SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 2011-05-31 19:47:06.018:INFO::Logging to STDERR via org.mortbay.log.StdErrLog [INFO] Executed tasks [INFO] [INFO] — build-helper-maven-plugin:1.5:add-source (add-jspc-source) @ hbase — [INFO] Source directory: /work/HBase451/hbase/target/jspc added. [INFO] [INFO] — build-helper-maven-plugin:1.5:add-source (add-package-info) @ hbase — [INFO] Source directory: /work/HBase451/hbase/target/generated-sources added. [INFO] [INFO] — jamon-maven-plugin:2.3.4:translate (default) @ hbase — [INFO] [INFO] — maven-remote-resources-plugin:1.1:process (default) @ hbase — [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on => 'false'. [INFO] Setting property: resource.loader => 'classpath'. [INFO] Setting property: resource.manager.logwhenfound => 'false'. [INFO] [INFO] — maven-resources-plugin:2.4:resources (default-resources) @ hbase — [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Copying 6 resources [INFO] Copying 3 resources [INFO] [INFO] — maven-antrun-plugin:1.3:run (default) @ hbase — [INFO] Executing tasks [INFO] Executed tasks [INFO] [INFO] — maven-compiler-plugin:2.0.2:compile (default-compile) @ hbase — [INFO] Compiling 8 source files to /work/HBase451/hbase/target/classes [INFO] [INFO] — maven-resources-plugin:2.4:testResources (default-testResources) @ hbase — [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 3 resources [INFO] [INFO] — maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase — [INFO] Compiling 1 source file to /work/HBase451/hbase/target/test-classes [INFO] [INFO] — maven-surefire-plugin:2.4.3:test (default-test) @ hbase — [INFO] Surefire report directory: /work/HBase451/hbase/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.136 sec Results : Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:36.596s [INFO] Finished at: Tue May 31 19:49:35 PDT 2011 [INFO] Final Memory: 19M/81M [INFO] ------------------------------------------------------------------------ =================================================================================================
          Hide
          Subbu M Iyer added a comment -

          Hi Ted,

          Can you send me your complete failure list?

          Show
          Subbu M Iyer added a comment - Hi Ted, Can you send me your complete failure list?
          Hide
          Ted Yu added a comment -

          Here is the list:
          http://pastebin.com/M5RS9Rib

          Show
          Ted Yu added a comment - Here is the list: http://pastebin.com/M5RS9Rib
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HConstants.java, line 174

          > <https://reviews.apache.org/r/849/diff/1/?file=20288#file20288line174>

          >

          > Copy/paste error.

          Fixed.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 211

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line211>

          >

          > When would this be used?

          Currently used from test classes. can remove it after validation.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 512

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line512>

          >

          > This should not be allowed? Are you not going to mess up stuff if table gets changed on an HRI?

          Yup. totally. removed it.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 547

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line547>

          >

          > Should these be deprecated rather than removed? Even if we returned a null only?

          Hmm. Any advantage of keeping it deprecated and returning NULL versus totally slaughtering? I can bring it back on.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 560

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line560>

          >

          > You could just do an equals since its boolean... Bytes.equals...

          Good. Done.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 568

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line568>

          >

          > ditto

          Good. Done.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 574

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line574>

          >

          > This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member.

          Not sure I get it. We do need support for isMetaRegion API call to determine if the Region represented by this HRI is a meta one. How shall we do it?

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 20

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line20>

          >

          > Put this into a migration subpackage?

          Could not agree more on this. I shall create a migration package *.hbase.migration and move this there.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 42

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line42>

          >

          > Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI.

          hmm.. makes sense. so you want to remove it from the class name.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 48

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line48>

          >

          > This is from elsewhere? Does it belong here?

          Its from old school HRI "as is".

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 600

          > <https://reviews.apache.org/r/849/diff/1/?file=20291#file20291line600>

          >

          > I don't think we should do this by default. What if KV has a value of MBs?

          yup. good call. removed it.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 250

          > <https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line250>

          >

          > What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style?

          Yup agreed. Need to think about how to handle this. any ideas? We might probably need to read the META rows using old school HRI till a failure (IOException trying to serialize a new HRI into old HRI) and assume that the rest are all processed?

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 276

          > <https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line276>

          >

          > You want to leave these in place?

          Nope. slipped through the cracks

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 638

          > <https://reviews.apache.org/r/849/diff/1/?file=20296#file20296line638>

          >

          > What a crazy way of getting HTDs.

          You want to rename? suggestions?

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 216

          > <https://reviews.apache.org/r/849/diff/1/?file=20301#file20301line216>

          >

          > Do we need these last two? Can't we use this first method to get the latter?

          We definitely can. Thought we can avoid iterating over the loop esp if we have large number of tables in the cluster.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java, line 2179

          > <https://reviews.apache.org/r/849/diff/1/?file=20302#file20302line2179>

          >

          > This method belongs in master since its going to run the migration?

          Yup moved it to Master.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 421

          > <https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line421>

          >

          > Close the file?

          Yup. Done

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 473

          > <https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line473>

          >

          > I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo?

          Let me check this. I have seen this creating in tmp and renaming it later pattern.

          • Subbu

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/849/#review749
          -----------------------------------------------------------

          On 2011-06-02 20:52:15, Michael Stack wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/849/

          -----------------------------------------------------------

          (Updated 2011-06-02 20:52:15)

          Review request for hbase.

          Summary

          -------

          Posting for Subbu See issue for his description of change.

          This addresses bug HBASE-451.

          https://issues.apache.org/jira/browse/HBASE-451

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0

          src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822

          src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0

          src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30

          src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b

          src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39

          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4

          src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17

          src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20

          src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520

          src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47

          src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150

          src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483

          src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c

          src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154

          src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad

          src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6

          src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0

          src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc

          src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df

          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803

          src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97

          src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791

          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

          src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815

          src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703

          src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc

          src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509

          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe

          src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb

          src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a

          src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3

          src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6

          src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0

          src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997

          src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767

          src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788

          src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507

          src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162

          src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371

          src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912

          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1

          src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f

          src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95

          src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a

          src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda

          src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055

          Diff: https://reviews.apache.org/r/849/diff

          Testing

          -------

          Thanks,

          Michael

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HConstants.java, line 174 > < https://reviews.apache.org/r/849/diff/1/?file=20288#file20288line174 > > > Copy/paste error. Fixed. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 211 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line211 > > > When would this be used? Currently used from test classes. can remove it after validation. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 512 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line512 > > > This should not be allowed? Are you not going to mess up stuff if table gets changed on an HRI? Yup. totally. removed it. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 547 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line547 > > > Should these be deprecated rather than removed? Even if we returned a null only? Hmm. Any advantage of keeping it deprecated and returning NULL versus totally slaughtering? I can bring it back on. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 560 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line560 > > > You could just do an equals since its boolean... Bytes.equals... Good. Done. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 568 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line568 > > > ditto Good. Done. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 574 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line574 > > > This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member. Not sure I get it. We do need support for isMetaRegion API call to determine if the Region represented by this HRI is a meta one. How shall we do it? On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 20 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line20 > > > Put this into a migration subpackage? Could not agree more on this. I shall create a migration package *.hbase.migration and move this there. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 42 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line42 > > > Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI. hmm.. makes sense. so you want to remove it from the class name. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 48 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line48 > > > This is from elsewhere? Does it belong here? Its from old school HRI "as is". On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 600 > < https://reviews.apache.org/r/849/diff/1/?file=20291#file20291line600 > > > I don't think we should do this by default. What if KV has a value of MBs? yup. good call. removed it. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 250 > < https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line250 > > > What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style? Yup agreed. Need to think about how to handle this. any ideas? We might probably need to read the META rows using old school HRI till a failure (IOException trying to serialize a new HRI into old HRI) and assume that the rest are all processed? On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 276 > < https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line276 > > > You want to leave these in place? Nope. slipped through the cracks On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 638 > < https://reviews.apache.org/r/849/diff/1/?file=20296#file20296line638 > > > What a crazy way of getting HTDs. You want to rename? suggestions? On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 216 > < https://reviews.apache.org/r/849/diff/1/?file=20301#file20301line216 > > > Do we need these last two? Can't we use this first method to get the latter? We definitely can. Thought we can avoid iterating over the loop esp if we have large number of tables in the cluster. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java, line 2179 > < https://reviews.apache.org/r/849/diff/1/?file=20302#file20302line2179 > > > This method belongs in master since its going to run the migration? Yup moved it to Master. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 421 > < https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line421 > > > Close the file? Yup. Done On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 473 > < https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line473 > > > I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo? Let me check this. I have seen this creating in tmp and renaming it later pattern. Subbu ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/#review749 ----------------------------------------------------------- On 2011-06-02 20:52:15, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/ ----------------------------------------------------------- (Updated 2011-06-02 20:52:15) Review request for hbase. Summary ------- Posting for Subbu See issue for his description of change. This addresses bug HBASE-451 . https://issues.apache.org/jira/browse/HBASE-451 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800 src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0 src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822 src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0 src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30 src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4 src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17 src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20 src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520 src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47 src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150 src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483 src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6 src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0 src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803 src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97 src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815 src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703 src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192 src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6 src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3 src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6 src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0 src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997 src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767 src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788 src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507 src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162 src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371 src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2 src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912 src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1 src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95 src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0 src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2 src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055 Diff: https://reviews.apache.org/r/849/diff Testing ------- Thanks, Michael
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 211

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line211>

          >

          > When would this be used?

          Subbu Iyer wrote:

          Currently used from test classes. can remove it after validation.

          Its ok. Just flag it as used from tests w/ a comment.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 547

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line547>

          >

          > Should these be deprecated rather than removed? Even if we returned a null only?

          Subbu Iyer wrote:

          Hmm. Any advantage of keeping it deprecated and returning NULL versus totally slaughtering? I can bring it back on.

          We like to deprecate for one major release before removing. Can you deprecate and then actually read the FS to get the actual HTD and return it? You can deprecate it and say its EXPENSIVE to call.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 574

          > <https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line574>

          >

          > This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member.

          Subbu Iyer wrote:

          Not sure I get it. We do need support for isMetaRegion API call to determine if the Region represented by this HRI is a meta one. How shall we do it?

          Well, IIUC, isMetaTable currently returns TRUE if either ROOT or .META. I was just flagging that you could introduce strange bugs if you change how this method works. if you need a method for is.META. then add new one I'd suggest (though it'll clash with this one).

          Or, hang on, perhaps I am misleading you. I am confusing isMetaTable with isMetaRegion? The latter is the one that returns true if ROOT or .META. isMetaTable should be true only if its .META. table.

          Sorry about that.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 20

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line20>

          >

          > Put this into a migration subpackage?

          Subbu Iyer wrote:

          Could not agree more on this. I shall create a migration package *.hbase.migration and move this there.

          Good.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 42

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line42>

          >

          > Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI.

          Subbu Iyer wrote:

          hmm.. makes sense. so you want to remove it from the class name.

          Up to you. I'd remove 'ForMigration' from class name or replace it with 090x so its HRegionInfo090x so it indicates its old style HRI. Up to you.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 48

          > <https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line48>

          >

          > This is from elsewhere? Does it belong here?

          Subbu Iyer wrote:

          Its from old school HRI "as is".

          Ok

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 250

          > <https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line250>

          >

          > What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style?

          Subbu Iyer wrote:

          Yup agreed. Need to think about how to handle this. any ideas? We might probably need to read the META rows using old school HRI till a failure (IOException trying to serialize a new HRI into old HRI) and assume that the rest are all processed?

          I think our suggestion of failed deserialization of old type HRI triggers attempt at deserializing it as new-type a good one.

          Only other thought I had was of serializing new type with some MAGIC bytes first, magic that you discarded when undoing it.... but I think doing this will probably make things way more complicated.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 638

          > <https://reviews.apache.org/r/849/diff/1/?file=20296#file20296line638>

          >

          > What a crazy way of getting HTDs.

          Subbu Iyer wrote:

          You want to rename? suggestions?

          No. You removed it which is good. I was just commenting on how crazy our old implementation was.

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 216

          > <https://reviews.apache.org/r/849/diff/1/?file=20301#file20301line216>

          >

          > Do we need these last two? Can't we use this first method to get the latter?

          Subbu Iyer wrote:

          We definitely can. Thought we can avoid iterating over the loop esp if we have large number of tables in the cluster.

          Up to you. Just seems a little gratuitous adding these three methods. If you can cut out one at least....?

          On 2011-06-02 23:07:01, Michael Stack wrote:

          > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 473

          > <https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line473>

          >

          > I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo?

          Subbu Iyer wrote:

          Let me check this. I have seen this creating in tmp and renaming it later pattern.

          Yes. Odd things can happen when regionserver crashes mid-writing. Zero files that make no sense. Table schema would be pretty bad to lose.

          • Michael

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/849/#review749
          -----------------------------------------------------------

          On 2011-06-02 20:52:15, Michael Stack wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/849/

          -----------------------------------------------------------

          (Updated 2011-06-02 20:52:15)

          Review request for hbase.

          Summary

          -------

          Posting for Subbu See issue for his description of change.

          This addresses bug HBASE-451.

          https://issues.apache.org/jira/browse/HBASE-451

          Diffs

          -----

          src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c

          src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d

          src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION

          src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800

          src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0

          src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197

          src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822

          src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155

          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0

          src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd

          src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30

          src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b

          src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d

          src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39

          src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8

          src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7

          src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac

          src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4

          src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17

          src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20

          src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520

          src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47

          src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150

          src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483

          src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c

          src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893

          src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154

          src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248

          src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad

          src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788

          src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6

          src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947

          src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108

          src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0

          src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc

          src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df

          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803

          src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97

          src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791

          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788

          src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815

          src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703

          src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc

          src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509

          src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe

          src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION

          src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6

          src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb

          src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a

          src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3

          src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72

          src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6

          src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0

          src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997

          src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767

          src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788

          src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507

          src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162

          src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45

          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b

          src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e

          src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371

          src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2

          src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912

          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1

          src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f

          src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154

          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95

          src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a

          src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda

          src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2

          src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055

          Diff: https://reviews.apache.org/r/849/diff

          Testing

          -------

          Thanks,

          Michael

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 211 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line211 > > > When would this be used? Subbu Iyer wrote: Currently used from test classes. can remove it after validation. Its ok. Just flag it as used from tests w/ a comment. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 547 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line547 > > > Should these be deprecated rather than removed? Even if we returned a null only? Subbu Iyer wrote: Hmm. Any advantage of keeping it deprecated and returning NULL versus totally slaughtering? I can bring it back on. We like to deprecate for one major release before removing. Can you deprecate and then actually read the FS to get the actual HTD and return it? You can deprecate it and say its EXPENSIVE to call. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 574 > < https://reviews.apache.org/r/849/diff/1/?file=20289#file20289line574 > > > This is incorrect. Should be isMetaTable() or isRootRegion... a 'meta' region is a .META. or ROOT member. Subbu Iyer wrote: Not sure I get it. We do need support for isMetaRegion API call to determine if the Region represented by this HRI is a meta one. How shall we do it? Well, IIUC, isMetaTable currently returns TRUE if either ROOT or .META. I was just flagging that you could introduce strange bugs if you change how this method works. if you need a method for is.META. then add new one I'd suggest (though it'll clash with this one). Or, hang on, perhaps I am misleading you. I am confusing isMetaTable with isMetaRegion? The latter is the one that returns true if ROOT or .META. isMetaTable should be true only if its .META. table. Sorry about that. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 20 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line20 > > > Put this into a migration subpackage? Subbu Iyer wrote: Could not agree more on this. I shall create a migration package *.hbase.migration and move this there. Good. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 42 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line42 > > > Oh, so what is this? Its old-school HRI? If so, call it HRI090? Or if you have it in a subpackage migration, it can have same name and just be fully specified whereever it is used.. the migration subpackage will make it explicity taht this is not same as new HRI. Subbu Iyer wrote: hmm.. makes sense. so you want to remove it from the class name. Up to you. I'd remove 'ForMigration' from class name or replace it with 090x so its HRegionInfo090x so it indicates its old style HRI. Up to you. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java, line 48 > < https://reviews.apache.org/r/849/diff/1/?file=20290#file20290line48 > > > This is from elsewhere? Does it belong here? Subbu Iyer wrote: Its from old school HRI "as is". Ok On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 250 > < https://reviews.apache.org/r/849/diff/1/?file=20292#file20292line250 > > > What if we crash half-way through migration? We should be able to deal w/ a .META. that is half the old style HRI and half the new style? Subbu Iyer wrote: Yup agreed. Need to think about how to handle this. any ideas? We might probably need to read the META rows using old school HRI till a failure (IOException trying to serialize a new HRI into old HRI) and assume that the rest are all processed? I think our suggestion of failed deserialization of old type HRI triggers attempt at deserializing it as new-type a good one. Only other thought I had was of serializing new type with some MAGIC bytes first, magic that you discarded when undoing it.... but I think doing this will probably make things way more complicated. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 638 > < https://reviews.apache.org/r/849/diff/1/?file=20296#file20296line638 > > > What a crazy way of getting HTDs. Subbu Iyer wrote: You want to rename? suggestions? No. You removed it which is good. I was just commenting on how crazy our old implementation was. On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 216 > < https://reviews.apache.org/r/849/diff/1/?file=20301#file20301line216 > > > Do we need these last two? Can't we use this first method to get the latter? Subbu Iyer wrote: We definitely can. Thought we can avoid iterating over the loop esp if we have large number of tables in the cluster. Up to you. Just seems a little gratuitous adding these three methods. If you can cut out one at least....? On 2011-06-02 23:07:01, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java, line 473 > < https://reviews.apache.org/r/849/diff/1/?file=20305#file20305line473 > > > I wonder if you have to create the file elsewhere and then do a rename to put it in place? Else if crash, could be half-written? Do we do this w/ .regioninfo? Subbu Iyer wrote: Let me check this. I have seen this creating in tmp and renaming it later pattern. Yes. Odd things can happen when regionserver crashes mid-writing. Zero files that make no sense. Table schema would be pretty bad to lose. Michael ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/#review749 ----------------------------------------------------------- On 2011-06-02 20:52:15, Michael Stack wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/849/ ----------------------------------------------------------- (Updated 2011-06-02 20:52:15) Review request for hbase. Summary ------- Posting for Subbu See issue for his description of change. This addresses bug HBASE-451 . https://issues.apache.org/jira/browse/HBASE-451 Diffs ----- src/main/java/org/apache/hadoop/hbase/HConstants.java bd4c64c src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 9502b1d src/main/java/org/apache/hadoop/hbase/HRegionInfoForMigration.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/KeyValue.java 7033800 src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java a25b0f0 src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java eb57197 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 7594822 src/main/java/org/apache/hadoop/hbase/client/HConnection.java f722155 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 94ee1a0 src/main/java/org/apache/hadoop/hbase/client/HTable.java c82e1dd src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 2734f30 src/main/java/org/apache/hadoop/hbase/client/UnmodifyableHRegionInfo.java 23e7a6b src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java d531b8d src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 4704c39 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 38914a8 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 5b4a4b7 src/main/java/org/apache/hadoop/hbase/master/HMaster.java b8489ac src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b22a3e4 src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java c98ed17 src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 5a8bb20 src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 6380520 src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 3d16e47 src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java dace150 src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java fcea483 src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java a963c6c src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 4029893 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java e5bd154 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 9ccf248 src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 072fd8d src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 21468ad src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java ba2daa9 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 0716788 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java 3def4b6 src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 1a87947 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3409108 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 39591a0 src/main/java/org/apache/hadoop/hbase/util/HMerge.java c07f8dc src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java 540d7df src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java 6d0c803 src/main/java/org/apache/hadoop/hbase/util/Writables.java 3e60f97 src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 4c58791 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java babd788 src/test/java/org/apache/hadoop/hbase/TestCompare.java bbac815 src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java 1f51703 src/test/java/org/apache/hadoop/hbase/TestSerialization.java 05f0efc src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 1105509 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java c3b23fe src/test/java/org/apache/hadoop/hbase/client/TestMetaMigration.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java db42192 src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java f0418d1 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 46ba184 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 18380c6 src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java 19397fb src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java e1eb02a src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java 04705c3 src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java bfa3c72 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java ada2af6 src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java ba87bc0 src/test/java/org/apache/hadoop/hbase/master/TestLoadBalancer.java d909997 src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java 2022767 src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java 1fef788 src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java e2f4507 src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 48fa162 src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 3b7c7e8 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java e106b45 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 516139b src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 40d352e src/test/java/org/apache/hadoop/hbase/regionserver/TestResettingCounters.java f092371 src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java ef8a4b2 src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java b85b912 src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java dbe5fb1 src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java 2cc197f src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java bcf9024 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java e2c158a src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java 5b95154 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 4de5b95 src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 3f9302a src/test/java/org/apache/hadoop/hbase/rest/model/TestTableRegionModel.java c02dfda src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a6bdbe0 src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java 3039df2 src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java 18cd055 Diff: https://reviews.apache.org/r/849/diff Testing ------- Thanks, Michael
          Hide
          Subbu M Iyer added a comment -

          Added second version of patch.

          Major changes include:
          1. Moved old school HRI to migration package and renamed it to HRegionInfo090x.
          2. Creating new .tableInfo in HDFS now follows the two step approach like .regionInfo. First create the file under .tmp and then rename it on successful create.
          3. Addressed most of the review call outs.

          Show
          Subbu M Iyer added a comment - Added second version of patch. Major changes include: 1. Moved old school HRI to migration package and renamed it to HRegionInfo090x. 2. Creating new .tableInfo in HDFS now follows the two step approach like .regionInfo. First create the file under .tmp and then rename it on successful create. 3. Addressed most of the review call outs.
          Hide
          Subbu M Iyer added a comment -

          I have one open issue on this.

          I need to address the cases when master crashes half way through the HRI migration process. I would like to address it as part of another JIRA as 1. I don't want to keep such a big change in my local with out commit for a long time 2) Also, it is easier for me to work on this issue off trunk rather than my local. (since I can be sure that everything else works and remains intact) 3) This will also help me keep my changes to a minimum and avoid a tedious patch work.

          Any thoughts/comments?

          Show
          Subbu M Iyer added a comment - I have one open issue on this. I need to address the cases when master crashes half way through the HRI migration process. I would like to address it as part of another JIRA as 1. I don't want to keep such a big change in my local with out commit for a long time 2) Also, it is easier for me to work on this issue off trunk rather than my local. (since I can be sure that everything else works and remains intact) 3) This will also help me keep my changes to a minimum and avoid a tedious patch work. Any thoughts/comments?
          Hide
          stack added a comment -

          @Subbu Please make a new blocker issue to deal w/ failed migration of .META. Lets get whats here in independent of that fixup.

          Show
          stack added a comment - @Subbu Please make a new blocker issue to deal w/ failed migration of .META. Lets get whats here in independent of that fixup.
          Hide
          stack added a comment -

          @Subbu I went to apply the patch to run tests but I'm getting way too many rejects. Have you git pulled recently? Mind taking a look? (Otherwise patch looks good).

          Show
          stack added a comment - @Subbu I went to apply the patch to run tests but I'm getting way too many rejects. Have you git pulled recently? Mind taking a look? (Otherwise patch looks good).
          Hide
          Subbu M Iyer added a comment -

          Stack: Sure. My git pull is one week old and looks like I will have to manually merge it (as I have more than 80+ files for this change) to latest and get a patch out of it. I am working on it.

          Show
          Subbu M Iyer added a comment - Stack: Sure. My git pull is one week old and looks like I will have to manually merge it (as I have more than 80+ files for this change) to latest and get a patch out of it. I am working on it.
          Hide
          Subbu M Iyer added a comment -

          I have created a new trunk snapshot last evening and applied my patch as well as resolved the conflicts.

          All tests have passed except the two below.
          1. TestReplication.queueFailover - I see a very informative note on this failure in the test as shown below, so I would
          think that this is not due to HBASE-451 changes.
          – WARNING this test sometimes fails because of HBASE-3515
          2. TestMasterFailover - Need to research more.

          Both these tests passed in my older snapshot build (which is one week+ old) and I will review it closer again.
          Also, a closer look at the failure log shows that these failures may not be due to HBASE-451 changes.

          Here are my priorities now:
          1. Address failures from your test run.
          2. Work on the issue of master crash during meta migration at startup.

          Show
          Subbu M Iyer added a comment - I have created a new trunk snapshot last evening and applied my patch as well as resolved the conflicts. All tests have passed except the two below. 1. TestReplication.queueFailover - I see a very informative note on this failure in the test as shown below, so I would think that this is not due to HBASE-451 changes. – WARNING this test sometimes fails because of HBASE-3515 2. TestMasterFailover - Need to research more. Both these tests passed in my older snapshot build (which is one week+ old) and I will review it closer again. Also, a closer look at the failure log shows that these failures may not be due to HBASE-451 changes. Here are my priorities now: 1. Address failures from your test run. 2. Work on the issue of master crash during meta migration at startup.
          Hide
          stack added a comment -

          I'm trying your patch over here Subbu. We can compare notes.

          Show
          stack added a comment - I'm trying your patch over here Subbu. We can compare notes.
          Hide
          stack added a comment -

          These failed for me Subbu:

          Running org.apache.hadoop.hbase.client.TestScannerTimeout
          Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 161.889 sec <<< FAILURE!

          Running org.apache.hadoop.hbase.util.TestMergeTool
          Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.363 sec <<< FAILURE!

          I retried these tests and indeed they are failing.

          Show
          stack added a comment - These failed for me Subbu: Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 161.889 sec <<< FAILURE! Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.363 sec <<< FAILURE! I retried these tests and indeed they are failing.
          Hide
          Subbu M Iyer added a comment -

          Stack:

          Can you attach the output/log for both the tests? I will try to troubleshoot the root cause.

          thanks

          Show
          Subbu M Iyer added a comment - Stack: Can you attach the output/log for both the tests? I will try to troubleshoot the root cause. thanks
          Hide
          Subbu M Iyer added a comment -

          Stack:

          I ran both these tests on my local version and both are passing.

          Please attach your output/error log so I can see what's happening. Also would help if u can attach your version of both the test files so I can compare if there is missing there.

          thanks

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hbase.client.TestScannerTimeout
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.164 sec

          Results :

          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:06.905s
          [INFO] Finished at: Fri Jun 10 18:30:43 PDT 2011
          [INFO] Final Memory: 18M/81M
          [INFO] ------------------------------------------------------------------------

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hbase.util.TestMergeTool
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.018 sec

          Results :

          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:21.650s
          [INFO] Finished at: Fri Jun 10 16:47:04 PDT 2011
          [INFO] Final Memory: 18M/81M
          [INFO] ------------------------------------------------------------------------

          Show
          Subbu M Iyer added a comment - Stack: I ran both these tests on my local version and both are passing. Please attach your output/error log so I can see what's happening. Also would help if u can attach your version of both the test files so I can compare if there is missing there. thanks ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.client.TestScannerTimeout Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.164 sec Results : Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:06.905s [INFO] Finished at: Fri Jun 10 18:30:43 PDT 2011 [INFO] Final Memory: 18M/81M [INFO] ------------------------------------------------------------------------ ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hbase.util.TestMergeTool Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.018 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:21.650s [INFO] Finished at: Fri Jun 10 16:47:04 PDT 2011 [INFO] Final Memory: 18M/81M [INFO] ------------------------------------------------------------------------
          Hide
          stack added a comment -

          They pass for you? If so, lets commit and figure it out later? You want me to commit the fourth draft?

          Show
          stack added a comment - They pass for you? If so, lets commit and figure it out later? You want me to commit the fourth draft?
          Hide
          Subbu M Iyer added a comment -

          Yes i would vote for a commit, so that, 1. It is easier to reproduce and fix for me 2) We can avoid another manual merge process as we won't be too much off from trunk.

          Let me know after a commit and I will work on these two tests if they are reproducible.

          thanks Stack.

          Show
          Subbu M Iyer added a comment - Yes i would vote for a commit, so that, 1. It is easier to reproduce and fix for me 2) We can avoid another manual merge process as we won't be too much off from trunk. Let me know after a commit and I will work on these two tests if they are reproducible. thanks Stack.
          Hide
          stack added a comment -

          @Subbu I just committed 'HBASE-451_-_Fourth_draft_support_for_removing_HTD_from_HRI.patch' (Thats, right?) Let me write the list to warn devs that TRUNK will be a little wonky for a while while we stabilize it. Good stuff.

          Show
          stack added a comment - @Subbu I just committed ' HBASE-451 _-_Fourth_draft_support_for_removing_HTD_from_HRI.patch' (Thats, right?) Let me write the list to warn devs that TRUNK will be a little wonky for a while while we stabilize it. Good stuff.
          Hide
          Subbu M Iyer added a comment -

          Thanks Stack. I will work on fixing failing tests if any.

          I also have a fix for the master crash half way through the META migration that will be addressed as part of another JIRA we created.

          Show
          Subbu M Iyer added a comment - Thanks Stack. I will work on fixing failing tests if any. I also have a fix for the master crash half way through the META migration that will be addressed as part of another JIRA we created.
          Hide
          stack added a comment -

          Subbu. TestAdmin is only test failing because of this commit. Reason its failing is because of our test of many concurrent table creates. I think the writing of the tableinfo file makes this all a little more complicated. I did a bit of cleanup in the attached patch. One byproduct is that now HRegion knows nothing of HTableDesc (we no longer try and create the tableinfo everytime we create a region – that seems a little wonky reaching up to the directory above....) What you think?

          Show
          stack added a comment - Subbu. TestAdmin is only test failing because of this commit. Reason its failing is because of our test of many concurrent table creates. I think the writing of the tableinfo file makes this all a little more complicated. I did a bit of cleanup in the attached patch. One byproduct is that now HRegion knows nothing of HTableDesc (we no longer try and create the tableinfo everytime we create a region – that seems a little wonky reaching up to the directory above....) What you think?
          Hide
          stack added a comment -

          Oh, patch fixes TestAdmin. Currently I'm testing to see if I broke others unit tests.

          Show
          stack added a comment - Oh, patch fixes TestAdmin. Currently I'm testing to see if I broke others unit tests.
          Hide
          stack added a comment -

          Hmm. Ignore the patch Subbu. I see that you read the tableinfo from the fs on each region open rather than have it passed in on the constructor. Why do the former than the latter? I'd think the latter better? The region host will have the current HTableDesc? If its passed in on the constructor, down the road the host can change it and the region will pick up the changes? Did you go the route of reading from the fs because you would not change the APIs for region creation? Thats a fair enough reason only I'd say these are all server-side APIs so no need for the deprecate cycle, etc.

          Show
          stack added a comment - Hmm. Ignore the patch Subbu. I see that you read the tableinfo from the fs on each region open rather than have it passed in on the constructor. Why do the former than the latter? I'd think the latter better? The region host will have the current HTableDesc? If its passed in on the constructor, down the road the host can change it and the region will pick up the changes? Did you go the route of reading from the fs because you would not change the APIs for region creation? Thats a fair enough reason only I'd say these are all server-side APIs so no need for the deprecate cycle, etc.
          Hide
          Subbu M Iyer added a comment -

          Stack: Is it really working with out HTD in createHRegion?

          We are creating the .tableInfo as part of createHRegion call and we won't be able to create a .tableInfo if HTD is not passed. Also with out .tableInfo & HTD during createHRegion most of the tests will fail.

          I understand it's kinda wonky to create a .tableInfo for every region creation, and my original idea was just to do it once at the time of create table. But there are lot of tests that just create a region (bypassing the create table and thus no HTD or tableinfo) and to support those cases I had to add this wonky support.

          Let me know how it looks with ur latest patch.

          Show
          Subbu M Iyer added a comment - Stack: Is it really working with out HTD in createHRegion? We are creating the .tableInfo as part of createHRegion call and we won't be able to create a .tableInfo if HTD is not passed. Also with out .tableInfo & HTD during createHRegion most of the tests will fail. I understand it's kinda wonky to create a .tableInfo for every region creation, and my original idea was just to do it once at the time of create table. But there are lot of tests that just create a region (bypassing the create table and thus no HTD or tableinfo) and to support those cases I had to add this wonky support. Let me know how it looks with ur latest patch.
          Hide
          stack added a comment -

          Stack: Is it really working with out HTD in createHRegion?

          No (smile).

          But I'm thinking that rather than have every region do a read of this file, instead we should pass in an HTD. This means changing the API for HRegion because it used to be part of passed in HRI. I can do this, np. Just say.

          Another i'm having is concurrent create of a table. The TestAdmin is failing doing this because the concurrent ops on the fs are running into interesting scenarios. I can work on this too, np.

          Show
          stack added a comment - Stack: Is it really working with out HTD in createHRegion? No (smile). But I'm thinking that rather than have every region do a read of this file, instead we should pass in an HTD. This means changing the API for HRegion because it used to be part of passed in HRI. I can do this, np. Just say. Another i'm having is concurrent create of a table. The TestAdmin is failing doing this because the concurrent ops on the fs are running into interesting scenarios. I can work on this too, np.
          Hide
          Subbu M Iyer added a comment -

          Hi Stack:

          I did see this problem with concurrent create and fixed it before, but looks like it slipped through the cracks. Here is a simple fix that addresses this and I have attached the patch. I validated that the TestAdmin passes on my latest trunk snapshot.

          Please let me know if any other tests are failing like TestMergeTool and TestSCannerTimeOut. I am kind of confused on why it failed it consistently before for you and it passed suddenly. May be some other networking blip/timeout or some thing?

          Regarding passing HTD as a variable in constructor: I did have one version of patch which had both the constructors and for some mysterious reasons (which I could never figure out), whenever I created a HREgion with HTD in the constructor, the PUTS will hang while trying to compute the MemStore size. (@ MemStore.internalAdd - this.size.addAndGet(s). I have spent like days on this before giving it up and sticking to only doing so at create time.

          By all means please feel free to make changes as you see fit and I am all open for it. I definitely want to take another pass at this once trunk is stabilized.

          thanks for your help/support on this.

          Show
          Subbu M Iyer added a comment - Hi Stack: I did see this problem with concurrent create and fixed it before, but looks like it slipped through the cracks. Here is a simple fix that addresses this and I have attached the patch. I validated that the TestAdmin passes on my latest trunk snapshot. Please let me know if any other tests are failing like TestMergeTool and TestSCannerTimeOut. I am kind of confused on why it failed it consistently before for you and it passed suddenly. May be some other networking blip/timeout or some thing? Regarding passing HTD as a variable in constructor: I did have one version of patch which had both the constructors and for some mysterious reasons (which I could never figure out), whenever I created a HREgion with HTD in the constructor, the PUTS will hang while trying to compute the MemStore size. (@ MemStore.internalAdd - this.size.addAndGet(s) . I have spent like days on this before giving it up and sticking to only doing so at create time. By all means please feel free to make changes as you see fit and I am all open for it. I definitely want to take another pass at this once trunk is stabilized. thanks for your help/support on this.
          Hide
          Subbu M Iyer added a comment -

          Please use TestAdmin1.patch as I accidentally included my pom changes.

          For some reason the latest trunk is failing for me when I do a compile. I am still using old POM to make thins work. Any ideas why?

          Show
          Subbu M Iyer added a comment - Please use TestAdmin1.patch as I accidentally included my pom changes. For some reason the latest trunk is failing for me when I do a compile. I am still using old POM to make thins work. Any ideas why?
          Hide
          stack added a comment -

          @Subbu You are current?

          Show
          stack added a comment - @Subbu You are current?
          Hide
          stack added a comment -

          The patch fixes TestAdmin so I committed it. Let me look at changing how HTD gets into HR (thanks for headsup about the Puts fail).

          Show
          stack added a comment - The patch fixes TestAdmin so I committed it. Let me look at changing how HTD gets into HR (thanks for headsup about the Puts fail).
          Hide
          Subbu M Iyer added a comment -

          Stack: can you please confirm that all tests are now passing?

          Also regarding my build, this is what I get when I compile goal from Idea.

          I simply rolled back my pom to a previous version (Revision 1099198) to make things work. Any ideas how I can fix this?

          =========================================================================================

          /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBaseTrunk0615/hbase/pom.xml compile
          [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions.
          [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions.
          [INFO] Scanning for projects...
          [WARNING]
          [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT
          [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 574, column 15
          [WARNING]
          [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
          [WARNING]
          [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
          [WARNING]
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building HBase 0.91.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml

          [WARNING] Could not transfer metadata org.apache.maven.plugins:maven-eclipse-plugin/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org
          [INFO]
          [INFO] — maven-antrun-plugin:1.6:run (generate) @ hbase —
          Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom

          Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom

          Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom

          Downloading: http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/2.0.5/plexus-utils-2.0.5.pom

          Downloading: http://repo1.maven.org/maven2/org/apache/ant/ant-nodeps/1.8.1/ant-nodeps-1.8.1.pom

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2.576s
          [INFO] Finished at: Wed Jun 15 12:03:30 PDT 2011
          [INFO] Final Memory: 5M/81M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (generate) on project hbase: Execution generate of goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.6 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.maven.plugins:maven-antrun-plugin:jar:1.6 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-api:jar:2.0.11: Could not transfer artifact org.apache.maven:maven-plugin-api:pom:2.0.11 from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org: Unknown host repo1.maven.org -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

          Process finished with exit code 1

          Show
          Subbu M Iyer added a comment - Stack: can you please confirm that all tests are now passing? Also regarding my build, this is what I get when I compile goal from Idea. I simply rolled back my pom to a previous version (Revision 1099198) to make things work. Any ideas how I can fix this? ========================================================================================= /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Dclassworlds.conf=/work/maven/apache-maven-3.0.2/bin/m2.conf -Dmaven.home=/work/maven/apache-maven-3.0.2 -Dfile.encoding=MacRoman -classpath /work/maven/apache-maven-3.0.2/boot/plexus-classworlds-2.4.jar org.codehaus.classworlds.Launcher --no-plugin-registry --fail-fast --no-plugin-updates --strict-checksums --update-snapshots -f /work/HBaseTrunk0615/hbase/pom.xml compile [WARNING] Command line option -npu is deprecated and will be removed in future Maven versions. [WARNING] Command line option -npr is deprecated and will be removed in future Maven versions. [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-eclipse-plugin is missing. @ line 574, column 15 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase 0.91.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-eclipse-plugin/maven-metadata.xml [WARNING] Could not transfer metadata org.apache.maven.plugins:maven-eclipse-plugin/maven-metadata.xml from/to central ( http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org [INFO] [INFO] — maven-antrun-plugin:1.6:run (generate) @ hbase — Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.11/maven-plugin-api-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-project/2.0.11/maven-project-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/apache/maven/maven-artifact/2.0.11/maven-artifact-2.0.11.pom Downloading: http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/2.0.5/plexus-utils-2.0.5.pom Downloading: http://repo1.maven.org/maven2/org/apache/ant/ant-nodeps/1.8.1/ant-nodeps-1.8.1.pom [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.576s [INFO] Finished at: Wed Jun 15 12:03:30 PDT 2011 [INFO] Final Memory: 5M/81M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (generate) on project hbase: Execution generate of goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.6 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.maven.plugins:maven-antrun-plugin:jar:1.6 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-api:jar:2.0.11: Could not transfer artifact org.apache.maven:maven-plugin-api:pom:2.0.11 from/to central ( http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org: Unknown host repo1.maven.org -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException Process finished with exit code 1
          Hide
          stack added a comment -

          Just did a test and now all pass (one at least was broke but not your fault). Not sure whats going on w/ your build. What version of mvn?

          Show
          stack added a comment - Just did a test and now all pass (one at least was broke but not your fault). Not sure whats going on w/ your build. What version of mvn?
          Hide
          stack added a comment -

          First cut at passing HTD rather than have HRegion read it from fs every time on construction.

          Compiles. Haven't tried all tests yet. Posting in case you want to have a look Subbu, see if direction looks right.

          Show
          stack added a comment - First cut at passing HTD rather than have HRegion read it from fs every time on construction. Compiles. Haven't tried all tests yet. Posting in case you want to have a look Subbu, see if direction looks right.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #1976 (See https://builds.apache.org/job/HBase-TRUNK/1976/)

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #1976 (See https://builds.apache.org/job/HBase-TRUNK/1976/ )
          Hide
          Subbu M Iyer added a comment -

          Stack:

          Checked your patch and this is exactly what I was trying to fight with. Hopefully it works this time. A simple test would be to run the TestColumnPrefixFilter and see if it succeeds. It used to hang around the PUT and it is a positive sign if this one succeeds.

          Let me know your findings. My MVN version is 3.0.2 and as I mentioned my problem occurs only with newer version of pom.

          Show
          Subbu M Iyer added a comment - Stack: Checked your patch and this is exactly what I was trying to fight with. Hopefully it works this time. A simple test would be to run the TestColumnPrefixFilter and see if it succeeds. It used to hang around the PUT and it is a positive sign if this one succeeds. Let me know your findings. My MVN version is 3.0.2 and as I mentioned my problem occurs only with newer version of pom.
          Hide
          stack added a comment -

          All but three pass (TestColumnPrefixFilter passes). Let me clean up this patch and fix outstanding tests.

          Show
          stack added a comment - All but three pass (TestColumnPrefixFilter passes). Let me clean up this patch and fix outstanding tests.
          Hide
          Ted Yu added a comment -

          I ran tests based on pass_htd_on_region_construction.txt and only got one failure:

          testSizes(org.apache.hadoop.hbase.io.TestHeapSize)  Time elapsed: 0.023 sec  <<< FAILURE!
          junit.framework.AssertionFailedError: expected:<296> but was:<304>
                  at junit.framework.Assert.fail(Assert.java:47)
                  at junit.framework.Assert.failNotEquals(Assert.java:283)
                  at junit.framework.Assert.assertEquals(Assert.java:64)
                  at junit.framework.Assert.assertEquals(Assert.java:130)
                  at junit.framework.Assert.assertEquals(Assert.java:136)
                  at org.apache.hadoop.hbase.io.TestHeapSize.testSizes(TestHeapSize.java:309)
          
          Show
          Ted Yu added a comment - I ran tests based on pass_htd_on_region_construction.txt and only got one failure: testSizes(org.apache.hadoop.hbase.io.TestHeapSize) Time elapsed: 0.023 sec <<< FAILURE! junit.framework.AssertionFailedError: expected:<296> but was:<304> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:130) at junit.framework.Assert.assertEquals(Assert.java:136) at org.apache.hadoop.hbase.io.TestHeapSize.testSizes(TestHeapSize.java:309)
          Hide
          stack added a comment -

          Thanks Ted. I'll fix that one. I've a new patch coming though... one sec. (Its taken me a while).

          Show
          stack added a comment - Thanks Ted. I'll fix that one. I've a new patch coming though... one sec. (Its taken me a while).
          Hide
          stack added a comment -

          New patch. Gathers up the getting and setting of TableDescriptors under new TableDescriptors interface; we supply an implemenation that goes against FS (this does reading/writing of the .tableinfo files).

          Any chance of a quick review Subbu?

          Tests passing but they are not finished.

          Show
          stack added a comment - New patch. Gathers up the getting and setting of TableDescriptors under new TableDescriptors interface; we supply an implemenation that goes against FS (this does reading/writing of the .tableinfo files). Any chance of a quick review Subbu? Tests passing but they are not finished.
          Hide
          Jean-Daniel Cryans added a comment -

          +1, kinda hard to review because of all the side effects of passing the HTD but Stack walked me through.

          Show
          Jean-Daniel Cryans added a comment - +1, kinda hard to review because of all the side effects of passing the HTD but Stack walked me through.
          Hide
          stack added a comment -

          OK. I committed patch after a few little fixes for a few failing tests found.

          Show
          stack added a comment - OK. I committed patch after a few little fixes for a few failing tests found.
          Hide
          stack added a comment -

          Let me resolve this issue. We've had enough fun and games in here, please its over 3 years old.

          Show
          stack added a comment - Let me resolve this issue. We've had enough fun and games in here, please its over 3 years old.
          Hide
          stack added a comment -

          Resolving. Thanks for nice work Subbu.

          Show
          stack added a comment - Resolving. Thanks for nice work Subbu.
          Hide
          Andrew Purtell added a comment -

          TestFSTableDescriptors.testHTableDescriptors has been broken since revision 1138120 ("HBASE-451 Remove HTableDescriptor from HRegionInfo – part 2, some cleanup")

          Show
          Andrew Purtell added a comment - TestFSTableDescriptors.testHTableDescriptors has been broken since revision 1138120 (" HBASE-451 Remove HTableDescriptor from HRegionInfo – part 2, some cleanup")
          Hide
          Gary Helmling added a comment -

          TestTableMapReduce seems to be hanging consistently on a master abort. From the logs, it looks like it's related to these changes, particularly:

          2011-06-24 12:30:29,159 INFO  [PRI IPC Server handler 9 on 42359] regionserver.HRegionServer(2300): Received request to open 25 region(s)
          2011-06-24 12:30:29,159 INFO  [PRI IPC Server handler 9 on 42359] regionserver.HRegionServer(2283): Received request to open region: mrtest,,1308943786187.294c624b4dca488152adc33fb47ffba6.
          2011-06-24 12:30:29,166 DEBUG [PRI IPC Server handler 9 on 42359] util.FSUtils(943): Exception during readTableDecriptor. Current table name = mrtest
          java.io.IOException: Cannot open filename /user/ghelmling/mrtest/.tableinfo
                  at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1527)
                  at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1518)
                  at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:384)
                  at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178)
                  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
                  at org.apache.hadoop.hbase.util.FSUtils.getTableDescriptor(FSUtils.java:953)
                  at org.apache.hadoop.hbase.util.FSUtils.getTableDescriptor(FSUtils.java:938)
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:130)
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
                  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
          2011-06-24 12:30:29,169 FATAL [ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0] master.HMaster(1198): Uncaught exception in ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0
          java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.TableExistsException: No descriptor for mrtest
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:134)
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
                  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
          
                  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1088)
                  at org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1654)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.TableExistsException: No descriptor for mrtest
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:134)
                  at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
                  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
          
                  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:837)
                  at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141)
                  at $Proxy10.openRegions(Unknown Source)
                  at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:422)
                  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1075)
                  ... 4 more
          2011-06-24 12:30:29,170 INFO  [ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0] master.HMaster(1323): Aborting
          2011-06-24 12:30:29,263 INFO  [ghelmling-laptop.local,52891,1308943821042.splitLogManagerTimeoutMonitor] hbase.Chore(79): ghelmling-laptop.local,52891,1308943821042.splitLogManagerTimeoutMonitor exiting
          

          Eventually this causes a region server to spin when the cluster attempts to shutdown:

          2011-06-24 12:31:09,767 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry
          2011-06-24 12:31:10,768 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry
          2011-06-24 12:31:11,769 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry
          2011-06-24 12:31:12,769 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry
          
          Show
          Gary Helmling added a comment - TestTableMapReduce seems to be hanging consistently on a master abort. From the logs, it looks like it's related to these changes, particularly: 2011-06-24 12:30:29,159 INFO [PRI IPC Server handler 9 on 42359] regionserver.HRegionServer(2300): Received request to open 25 region(s) 2011-06-24 12:30:29,159 INFO [PRI IPC Server handler 9 on 42359] regionserver.HRegionServer(2283): Received request to open region: mrtest,,1308943786187.294c624b4dca488152adc33fb47ffba6. 2011-06-24 12:30:29,166 DEBUG [PRI IPC Server handler 9 on 42359] util.FSUtils(943): Exception during readTableDecriptor. Current table name = mrtest java.io.IOException: Cannot open filename /user/ghelmling/mrtest/.tableinfo at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1527) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1518) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:384) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:178) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356) at org.apache.hadoop.hbase.util.FSUtils.getTableDescriptor(FSUtils.java:953) at org.apache.hadoop.hbase.util.FSUtils.getTableDescriptor(FSUtils.java:938) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:130) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065) 2011-06-24 12:30:29,169 FATAL [ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0] master.HMaster(1198): Uncaught exception in ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0 java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.TableExistsException: No descriptor for mrtest at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:134) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1088) at org.apache.hadoop.hbase.master.AssignmentManager$SingleServerBulkAssigner.run(AssignmentManager.java:1654) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.TableExistsException: No descriptor for mrtest at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:134) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:99) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2286) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegions(HRegionServer.java:2301) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:837) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:141) at $Proxy10.openRegions(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:422) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1075) ... 4 more 2011-06-24 12:30:29,170 INFO [ghelmling-laptop.local,52891,1308943821042-StartupBulkAssigner-0] master.HMaster(1323): Aborting 2011-06-24 12:30:29,263 INFO [ghelmling-laptop.local,52891,1308943821042.splitLogManagerTimeoutMonitor] hbase.Chore(79): ghelmling-laptop.local,52891,1308943821042.splitLogManagerTimeoutMonitor exiting Eventually this causes a region server to spin when the cluster attempts to shutdown: 2011-06-24 12:31:09,767 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry 2011-06-24 12:31:10,768 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry 2011-06-24 12:31:11,769 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry 2011-06-24 12:31:12,769 DEBUG [RegionServer:0;ghelmling-laptop.local,42359,1308943821244] regionserver.HRegionServer(1485): No master found; retry
          Hide
          stack added a comment -

          Sorry about that. I will fix monday

          Show
          stack added a comment - Sorry about that. I will fix monday
          Hide
          Ted Yu added a comment - - edited

          Looks like TestTableMapReduce doesn't create the table. From MultiRegionTable which TestTableMapReduce inherits:

              HRegion region = createNewHRegion(desc, startKey, endKey);
          

          I added test of whether table descriptor exists on HDFS in HRegion.createHRegion(). If it doesn't exist, I call FSUtils.createTableDescriptor().

          Now TestTableMapReduce and TestFSTableDescriptors both pass.
          TestDistributedLogSplitting and TestSplitTransactionOnCluster pass on my laptop as well.

          Show
          Ted Yu added a comment - - edited Looks like TestTableMapReduce doesn't create the table. From MultiRegionTable which TestTableMapReduce inherits: HRegion region = createNewHRegion(desc, startKey, endKey); I added test of whether table descriptor exists on HDFS in HRegion.createHRegion(). If it doesn't exist, I call FSUtils.createTableDescriptor(). Now TestTableMapReduce and TestFSTableDescriptors both pass. TestDistributedLogSplitting and TestSplitTransactionOnCluster pass on my laptop as well.
          Hide
          Ted Yu added a comment -

          Here is my addendum.
          There could be a cleaner way of detecting that table descriptor doesn't exist on HDFS. For the moment, I rely on TableExistsException.

          Show
          Ted Yu added a comment - Here is my addendum. There could be a cleaner way of detecting that table descriptor doesn't exist on HDFS. For the moment, I rely on TableExistsException.
          Hide
          Ted Yu added a comment -

          I am running test suite on Linux and TestSplitTransactionOnCluster passed.

          Show
          Ted Yu added a comment - I am running test suite on Linux and TestSplitTransactionOnCluster passed.
          Hide
          Ted Yu added a comment -

          TestDistributedLogSplitting hung on Linux.

          The second time I ran it on my laptop, I got:

          Failed tests: 
            testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)
          

          The first exception in output file was:

          2011-06-24 15:01:37,115 WARN  [PostOpenDeployTasks:1028785192] handler.OpenRegionHandler$PostOpenDeployTasksThread(221): Exception running postOpenDeployTasks; region=1028785192
          java.io.IOException: No server for -ROOT-
                  at org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaLocation(MetaEditor.java:149)
                  at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1405)
                  at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:218)
          

          Although there is no such exception in output on Linux.

          Show
          Ted Yu added a comment - TestDistributedLogSplitting hung on Linux. The second time I ran it on my laptop, I got: Failed tests: testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting) The first exception in output file was: 2011-06-24 15:01:37,115 WARN [PostOpenDeployTasks:1028785192] handler.OpenRegionHandler$PostOpenDeployTasksThread(221): Exception running postOpenDeployTasks; region=1028785192 java.io.IOException: No server for -ROOT- at org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaLocation(MetaEditor.java:149) at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1405) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:218) Although there is no such exception in output on Linux.
          Hide
          Gary Helmling added a comment -

          @Ted, good digging.

          I'm not very familiar with these changes, but it looks to me like the changes so far have tried to pull HTableDescription handling out of HRegion. So adding it back in to HRegion.createHRegion() may be a step back.

          I think I'd opt for trying to fix the tests to call FSUtils.createTableDescriptor() instead. Either in TestTableMapReduce.<init> or MultiRegionTable.preHBaseClusterSetup(), prior to creating the table regions. I think either of those would work. I wonder how many other HBaseTestCase subclasses may have problems as well.

          Show
          Gary Helmling added a comment - @Ted, good digging. I'm not very familiar with these changes, but it looks to me like the changes so far have tried to pull HTableDescription handling out of HRegion. So adding it back in to HRegion.createHRegion() may be a step back. I think I'd opt for trying to fix the tests to call FSUtils.createTableDescriptor() instead. Either in TestTableMapReduce.<init> or MultiRegionTable.preHBaseClusterSetup(), prior to creating the table regions. I think either of those would work. I wonder how many other HBaseTestCase subclasses may have problems as well.
          Hide
          Ted Yu added a comment -

          Thanks Gary for the reminder.
          I will upload addendum version 2 which creates table descriptor in MultiRegionTable.preHBaseClusterSetup().

          TestTableMapReduce passes:

          Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.72 sec
          Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.983 sec
          
          Show
          Ted Yu added a comment - Thanks Gary for the reminder. I will upload addendum version 2 which creates table descriptor in MultiRegionTable.preHBaseClusterSetup(). TestTableMapReduce passes: Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.72 sec Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.983 sec
          Hide
          Gary Helmling added a comment -

          @Ted,

          +1 from me on addendum v2, since the tests now pass.

          Show
          Gary Helmling added a comment - @Ted, +1 from me on addendum v2, since the tests now pass.
          Hide
          Ted Yu added a comment -

          On Linux, TestFSTableDescriptors fails at the following assertion:

              assertEquals(count * 2, htds.cachehits);
          

          The error was:

          java.lang.AssertionError: expected:<20> but was:<30>
          

          Stack should know how to fix the above.

          Show
          Ted Yu added a comment - On Linux, TestFSTableDescriptors fails at the following assertion: assertEquals(count * 2, htds.cachehits); The error was: java.lang.AssertionError: expected:<20> but was:<30> Stack should know how to fix the above.
          Hide
          Ted Yu added a comment -

          I committed addendum v2 into TRUNK.

          Show
          Ted Yu added a comment - I committed addendum v2 into TRUNK.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #1995 (See https://builds.apache.org/job/HBase-TRUNK/1995/)

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #1995 (See https://builds.apache.org/job/HBase-TRUNK/1995/ )
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2026 (See https://builds.apache.org/job/HBase-TRUNK/2026/)
          Fix broken TestScanMultipleVersions test – needed .tableinfo defined before test started... part of the repercussions from hbase-451

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2026 (See https://builds.apache.org/job/HBase-TRUNK/2026/ ) Fix broken TestScanMultipleVersions test – needed .tableinfo defined before test started... part of the repercussions from hbase-451
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2027 (See https://builds.apache.org/job/HBase-TRUNK/2027/)
          Fix broken TestMasterFailover test – needed .tableinfo defined before test started... part of the repercussions from hbase-451; I don't this fixes all TestMasterFailover failures but at least a few obvious failures

          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2027 (See https://builds.apache.org/job/HBase-TRUNK/2027/ ) Fix broken TestMasterFailover test – needed .tableinfo defined before test started... part of the repercussions from hbase-451; I don't this fixes all TestMasterFailover failures but at least a few obvious failures
          Hide
          stack added a comment -

          Reresolving this one. TestFSTableDescriptors passes now. Will address the remaining 451 issue, the fact that we return null when you do a get on descriptor over in the blocker Andrew opened for addressing this.

          Show
          stack added a comment - Reresolving this one. TestFSTableDescriptors passes now. Will address the remaining 451 issue, the fact that we return null when you do a get on descriptor over in the blocker Andrew opened for addressing this.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2034 (See https://builds.apache.org/job/HBase-TRUNK/2034/)
          HBASE-451 getTableInfoPath() should be called in FSUtils.getTableInfoModtime()

          tedyu :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2034 (See https://builds.apache.org/job/HBase-TRUNK/2034/ ) HBASE-451 getTableInfoPath() should be called in FSUtils.getTableInfoModtime() tedyu : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2035 (See https://builds.apache.org/job/HBase-TRUNK/2035/)
          HBASE-451 relax cachehits check in TestFSTableDescriptors

          tedyu :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2035 (See https://builds.apache.org/job/HBase-TRUNK/2035/ ) HBASE-451 relax cachehits check in TestFSTableDescriptors tedyu : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2038 (See https://builds.apache.org/job/HBase-TRUNK/2038/)
          HBASE-451 FSUtils.getTableInfoModtime() should check fs.listStatus() against null return

          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2038 (See https://builds.apache.org/job/HBase-TRUNK/2038/ ) HBASE-451 FSUtils.getTableInfoModtime() should check fs.listStatus() against null return tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2065 (See https://builds.apache.org/job/HBase-TRUNK/2065/)
          HBASE-4032 HBASE-451 improperly breaks public API HRegionInfo#getTableDesc

          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          • /hbase/trunk/CHANGES.txt
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2065 (See https://builds.apache.org/job/HBase-TRUNK/2065/ ) HBASE-4032 HBASE-451 improperly breaks public API HRegionInfo#getTableDesc tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java /hbase/trunk/CHANGES.txt /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
          Hide
          Benoit Sigoure added a comment -

          This change completely breaks RPC level compatibility because of this line added to HbaseObjectWritable:

          addToMap(HTableDescriptor[].class, code++);
          
          Show
          Benoit Sigoure added a comment - This change completely breaks RPC level compatibility because of this line added to HbaseObjectWritable : addToMap(HTableDescriptor[].class, code++);

            People

            • Assignee:
              Subbu M Iyer
              Reporter:
              Jim Kellerman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development