HBase
  1. HBase
  2. HBASE-2531

32-bit encoding of regionnames waaaaaaayyyyy too susceptible to hash clashes

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.90.0
    • Component/s: None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changes format of region name. Adds an md5 suffix. Suffix is now the name used as directory name in filesystem.

      Description

      Kannan tripped over two regionnames that hashed the same:

      Here is code demo'ing that his two names hash the same:

      package org;
      
      import org.apache.hadoop.hbase.util.Bytes;
      import org.apache.hadoop.hbase.util.JenkinsHash;
      
      
      public class Testing {
        public static void main(final String [] args) {
          System.out.println(encodeRegionName(Bytes.toBytes("test1,6838000000,1273541236167")));
          System.out.println(encodeRegionName(Bytes.toBytes("test1,0520100000,1273541610201")));
        }
      
        /**
         * @param regionName
         * @return the encodedName
         */
        public static int encodeRegionName(final byte [] regionName) {
          return Math.abs(JenkinsHash.getInstance().hash(regionName, regionName.length, 0));
        }
      }
      

      Need new encoding mechanism. Will need to migrate old regions to new schema.

      1. HBASE-2531_v2.patch
        34 kB
        Kannan Muthukkaruppan

        Issue Links

          Activity

          Hide
          stack added a comment -

          After some discussion up on IRC – todd, posix4e, jdcryans, kannan – and thought was that UUID would make the most sense.

          Show
          stack added a comment - After some discussion up on IRC – todd, posix4e, jdcryans, kannan – and thought was that UUID would make the most sense.
          Hide
          stack added a comment -

          This change would need to include runtime migration of old format to the new.

          Show
          stack added a comment - This change would need to include runtime migration of old format to the new.
          Hide
          Kannan Muthukkaruppan added a comment -

          Do we ever need to figure out the encoded name, given just the region name (<tablename>, <startkey>, <timestamp>)?

          If not, and say, we go with UUIDs to generate future encoded names, then is there any work needed to done for migration at all?

          After an upgrade, for a period, there'll be a mix of old and new format of encoded names, and when splits etc. happens and old region go away, gradually the system should only be left with new format names.

          Show
          Kannan Muthukkaruppan added a comment - Do we ever need to figure out the encoded name, given just the region name (<tablename>, <startkey>, <timestamp>)? If not, and say, we go with UUIDs to generate future encoded names, then is there any work needed to done for migration at all? After an upgrade, for a period, there'll be a mix of old and new format of encoded names, and when splits etc. happens and old region go away, gradually the system should only be left with new format names.
          Hide
          dhruba borthakur added a comment -

          Please allow me to interject myself in this conversation.

          It appears that UUID proposal will work. However, it always leaves the possibility of data corruption open. In the rare case when you might run two region servers on the same machine (if ever!), then we might get a chance of UUID collision, especially for a workload when region splitting occurs v frequently. The UUID approach also seems to imply that some sort of "migration of old format to new format" is required.

          Isn't it more elegant and easier if we do the following: "when a region server splits a region it needs to create a new name, It can come up with a random number as it currently does and then try to create a znode in zookeeper with that name, if it already exists, then the region server can generate a new name. if the znode does not exist, then it will create the znode before creating the new region with the region name. will that work?" This will let us avoid any need for "migration" while preventing any possibility of uuid collisions.

          Show
          dhruba borthakur added a comment - Please allow me to interject myself in this conversation. It appears that UUID proposal will work. However, it always leaves the possibility of data corruption open. In the rare case when you might run two region servers on the same machine (if ever!), then we might get a chance of UUID collision, especially for a workload when region splitting occurs v frequently. The UUID approach also seems to imply that some sort of "migration of old format to new format" is required. Isn't it more elegant and easier if we do the following: "when a region server splits a region it needs to create a new name, It can come up with a random number as it currently does and then try to create a znode in zookeeper with that name, if it already exists, then the region server can generate a new name. if the znode does not exist, then it will create the znode before creating the new region with the region name. will that work?" This will let us avoid any need for "migration" while preventing any possibility of uuid collisions.
          Hide
          Todd Lipcon added a comment -

          dhruba: I had suggested a similar thing on IRC last night, and people seemed to think that UUID would be easier to implement. The chances of multiple UUIDs generated in the same nanotime on the same machine seems pretty small, no?

          Show
          Todd Lipcon added a comment - dhruba: I had suggested a similar thing on IRC last night, and people seemed to think that UUID would be easier to implement. The chances of multiple UUIDs generated in the same nanotime on the same machine seems pretty small, no?
          Hide
          dhruba borthakur added a comment -

          I agree that the "chance of UUID collision is low". But is it really true (especially if this needs a "migration" as described in earlier comments in this JIRA) that the ZK approach is harder/complex to implement?

          Show
          dhruba borthakur added a comment - I agree that the "chance of UUID collision is low". But is it really true (especially if this needs a "migration" as described in earlier comments in this JIRA) that the ZK approach is harder/complex to implement?
          Hide
          Kannan Muthukkaruppan added a comment -

          Literature online seems to suggest that version 1 (timestamp based UUIDs) are unique for all practical purposes. [Also read that <<If two UUIDs are generated in sequence fast enough that the timestamp matches the previous UUID, the timestamp is incremented by 1.>>].

          Some references:

          http://stackoverflow.com/questions/703035/when-are-you-truly-forced-to-use-uuid-as-part-of-the-design
          http://blogs.msdn.com/oldnewthing/archive/2008/06/27/8659071.aspx

          Re: migration, in a ZK based approach, wouldn't moving the pre-existing list of encoded region names to ZK require some work?

          For UUID based approach, it seems like there shouldn't really be any migration work. Stack: can you confirm?

          I

          Show
          Kannan Muthukkaruppan added a comment - Literature online seems to suggest that version 1 (timestamp based UUIDs) are unique for all practical purposes. [Also read that <<If two UUIDs are generated in sequence fast enough that the timestamp matches the previous UUID, the timestamp is incremented by 1.>>] . Some references: http://stackoverflow.com/questions/703035/when-are-you-truly-forced-to-use-uuid-as-part-of-the-design http://blogs.msdn.com/oldnewthing/archive/2008/06/27/8659071.aspx Re: migration, in a ZK based approach, wouldn't moving the pre-existing list of encoded region names to ZK require some work? For UUID based approach, it seems like there shouldn't really be any migration work. Stack: can you confirm? I
          Hide
          stack added a comment -

          Bulk move of 0.20.5 issues into 0.21.0 after vote that we merge branch into TRUNK up on list.

          Show
          stack added a comment - Bulk move of 0.20.5 issues into 0.21.0 after vote that we merge branch into TRUNK up on list.
          Hide
          Kannan Muthukkaruppan added a comment -

          Stack: Pinging regarding my question on what migration work would be needed? (See my update @ 11/May/10 02:14 PM).

          Show
          Kannan Muthukkaruppan added a comment - Stack: Pinging regarding my question on what migration work would be needed? (See my update @ 11/May/10 02:14 PM).
          Hide
          stack added a comment -

          @Kannan As I see it, we just need to make sure that the system can work with both styles of naming; that it reads in the old stuff without issue and that at same time, for any new region created, we should use the UUID form writing new region directory names. Somehow, we also need to drop this notion of encoding the region name. Going forward it will not be needed since the UUID will actually be part of the region name.

          I agree that getting zk in the mix or even hdfs for that matter making region dirctory names complicates something that could be real simple if we use UUIDs.

          Show
          stack added a comment - @Kannan As I see it, we just need to make sure that the system can work with both styles of naming; that it reads in the old stuff without issue and that at same time, for any new region created, we should use the UUID form writing new region directory names. Somehow, we also need to drop this notion of encoding the region name. Going forward it will not be needed since the UUID will actually be part of the region name. I agree that getting zk in the mix or even hdfs for that matter making region dirctory names complicates something that could be real simple if we use UUIDs.
          Hide
          dhruba borthakur added a comment -

          > Going forward it will not be needed since the UUID will actually be part of the region name.

          @Stack, thanks for the info. If the UUD is part of the regionname, then does this mean that nothing needs to be done for this JIRA? If that is not the case and we want to do somethinkg for this JIRS, can you pl explain what needs to be done? Thanks.

          Show
          dhruba borthakur added a comment - > Going forward it will not be needed since the UUID will actually be part of the region name. @Stack, thanks for the info. If the UUD is part of the regionname, then does this mean that nothing needs to be done for this JIRA? If that is not the case and we want to do somethinkg for this JIRS, can you pl explain what needs to be done? Thanks.
          Hide
          stack added a comment -

          @Dhruba

          Some good stuff related to this issue came up on IRC this afternoon. In particular, we probably want region names to sort such that children of splits are inserted after their parent. One of the childs of a split will have same startkey as the parent, the one that is carrying the lower range of the split. The only differentiator is the 3rd part of the regionname where regionnames are formatted as

          <tablename> ',' <startkey> ',' <timestamp>
          

          .... and since it currently timestamp, the child will go into META after the parent (Things should still work even if child goes in before parent but of that I'm not certain).

          So, we probably want to keep this attribute of regionnames.

          This would seem to rule out UUID as the 3rd component of regionnames since they are effectively 'random' (version 1 was time-based but as Kannan points out, they are prefixed with MAC address and anyways, java doesn't do version 1 UUID).

          Show
          stack added a comment - @Dhruba Some good stuff related to this issue came up on IRC this afternoon. In particular, we probably want region names to sort such that children of splits are inserted after their parent. One of the childs of a split will have same startkey as the parent, the one that is carrying the lower range of the split. The only differentiator is the 3rd part of the regionname where regionnames are formatted as <tablename> ',' <startkey> ',' <timestamp> .... and since it currently timestamp, the child will go into META after the parent (Things should still work even if child goes in before parent but of that I'm not certain). So, we probably want to keep this attribute of regionnames. This would seem to rule out UUID as the 3rd component of regionnames since they are effectively 'random' (version 1 was time-based but as Kannan points out, they are prefixed with MAC address and anyways, java doesn't do version 1 UUID).
          Hide
          stack added a comment -

          ... continuing

          As to what needs to be done, at a minimum, we need to change the way we name dirs in the filesystem. Currently its done as follows

            /**
             * @param regionName
             * @return the encodedName
             */
            public static int encodeRegionName(final byte [] regionName) {
              return Math.abs(JenkinsHash.getInstance().hash(regionName, regionName.length, 0));
            }
          

          The minimally intrusive thing would be to change the above hashing to instead return a byte array or a String and have the function md5 or sha-1 the regionName so there is some relation between the regionname and hash, or just return a UUID, a product that cannot be related to the regionname. We'd then need to go through code base and make sure that everywhere we deal with the encoded name of the region, that we can handle BOTH the new style byte [] or String format and the old format int.

          Since we cannot derive the regionname from the UUID, we must be careful we do not misplace the UUID. We'd have to save it into the regions HRegionInfo object.

          md5/sha-1 would be superior because we can always go from regionname to the encoded name.

          I was thinking (and I think Kannan the same), that rather than timestamp alone as the 3rd component of the regionname, that rather we'd make it so the 3rd portion of the regionname serve two functions: its current one as differentiator between child and parent (see previous comment) but that this 3rd component would also be what we use for the region directory in the filesystem. Timestamp alone would not be enough. After this afternoon's IRC discussions, UUID isn't suitable. We'd have to tag on something extra. It could be an md5 of the startkey or it could just be jenkins hash of the startkey since likelihood of hash-of-startkey+timestamp would clash is unlikely.

          I liked this later option because you'd read the regionname and would be able to then easily find the region's dir in the filesystem.

          This would be a more intrusive change than the one above where we just change hash function.

          Show
          stack added a comment - ... continuing As to what needs to be done, at a minimum, we need to change the way we name dirs in the filesystem. Currently its done as follows /** * @param regionName * @ return the encodedName */ public static int encodeRegionName( final byte [] regionName) { return Math .abs(JenkinsHash.getInstance().hash(regionName, regionName.length, 0)); } The minimally intrusive thing would be to change the above hashing to instead return a byte array or a String and have the function md5 or sha-1 the regionName so there is some relation between the regionname and hash, or just return a UUID, a product that cannot be related to the regionname. We'd then need to go through code base and make sure that everywhere we deal with the encoded name of the region, that we can handle BOTH the new style byte [] or String format and the old format int. Since we cannot derive the regionname from the UUID, we must be careful we do not misplace the UUID. We'd have to save it into the regions HRegionInfo object. md5/sha-1 would be superior because we can always go from regionname to the encoded name. I was thinking (and I think Kannan the same), that rather than timestamp alone as the 3rd component of the regionname, that rather we'd make it so the 3rd portion of the regionname serve two functions: its current one as differentiator between child and parent (see previous comment) but that this 3rd component would also be what we use for the region directory in the filesystem. Timestamp alone would not be enough. After this afternoon's IRC discussions, UUID isn't suitable. We'd have to tag on something extra. It could be an md5 of the startkey or it could just be jenkins hash of the startkey since likelihood of hash-of-startkey+timestamp would clash is unlikely. I liked this later option because you'd read the regionname and would be able to then easily find the region's dir in the filesystem. This would be a more intrusive change than the one above where we just change hash function.
          Hide
          Kannan Muthukkaruppan added a comment -

          Writing out the details of one possible solution. Comments welcome.

          Old region names continue to have the format: <tablename>,<startkey>,<timestamp>. For region names in this format, the encoded name will continue to be the old JenkinsHash implementation.
          New region names have the format: <tablename>,<startkey>,<timestamp>,<dirname> where <dirname> is the md5 hash of the <tablename>,<startkey>,<timestamp> and will serve as encoded name/directory name in FS for the region.

          This preserves the property that child regions (splits) will have a region name that sorts higher than the parent.

          Search to determine what region serves a particular key is done today by building a key of the form:
          <tablename>,<searchkey>,99999999999999

          That's 14 9's. On our test cluster I noticed region names of the form: test1,0013440000,1273816773769. That's 13 digits for the timestamp part. Going forward, we'll have region names of the form: test1,<key>,<13digit-ts>,<md5hash>. But the 14 9's based search key would continue to work just fine even with the new format region names since '9' > ',' .

          Show
          Kannan Muthukkaruppan added a comment - Writing out the details of one possible solution. Comments welcome. Old region names continue to have the format: <tablename>,<startkey>,<timestamp>. For region names in this format, the encoded name will continue to be the old JenkinsHash implementation. New region names have the format: <tablename>,<startkey>,<timestamp>,<dirname> where <dirname> is the md5 hash of the <tablename>,<startkey>,<timestamp> and will serve as encoded name/directory name in FS for the region. This preserves the property that child regions (splits) will have a region name that sorts higher than the parent. Search to determine what region serves a particular key is done today by building a key of the form: <tablename>,<searchkey>,99999999999999 That's 14 9's. On our test cluster I noticed region names of the form: test1,0013440000,1273816773769. That's 13 digits for the timestamp part. Going forward, we'll have region names of the form: test1,<key>,<13digit-ts>,<md5hash>. But the 14 9's based search key would continue to work just fine even with the new format region names since '9' > ',' .
          Hide
          Todd Lipcon added a comment -

          Regardless of what we change it to, can I suggest that we add a new class RegionCode or RegionId which serves as a "typesafe id"? This makes it harder to have bugs where we mix up the various strings used to refer to a region (at the expense of a small amount of memory for the wrapper class).

          Show
          Todd Lipcon added a comment - Regardless of what we change it to, can I suggest that we add a new class RegionCode or RegionId which serves as a "typesafe id"? This makes it harder to have bugs where we mix up the various strings used to refer to a region (at the expense of a small amount of memory for the wrapper class).
          Hide
          stack added a comment -

          Yes to Todd suggestion.

          Kannan, I'm down w/ your suggesion except for bit where ',' is also the delimiter between timestamp and dirname. Use a '.' or something instead. Special meta region comparator code looks for the ',' characters dividing up the parts of a meta key doing sorting. The extra ',' will throw it off and you'll get a headache trying to sort out how this comparator works. it gets really interesting when meta splits. (though currently this is disabled).... for then you have meta regionnames that look like this: meta,TestTable,SOMESTARTKEY,TS,TS... then throw in fact that starkeys can be binary and my sense is that about now you feel a migrane coming on.

          I'm good w/ md5. 128 bits vs 160 bits for sha-1 (which seems overkill). Or we could keep jenkins hash – 32 bits – because and use timestamp+jenkins_hash naming dir. A collision is unlikely?

          Show
          stack added a comment - Yes to Todd suggestion. Kannan, I'm down w/ your suggesion except for bit where ',' is also the delimiter between timestamp and dirname. Use a '.' or something instead. Special meta region comparator code looks for the ',' characters dividing up the parts of a meta key doing sorting. The extra ',' will throw it off and you'll get a headache trying to sort out how this comparator works. it gets really interesting when meta splits. (though currently this is disabled).... for then you have meta regionnames that look like this: meta,TestTable,SOMESTARTKEY,TS,TS... then throw in fact that starkeys can be binary and my sense is that about now you feel a migrane coming on. I'm good w/ md5. 128 bits vs 160 bits for sha-1 (which seems overkill). Or we could keep jenkins hash – 32 bits – because and use timestamp+jenkins_hash naming dir. A collision is unlikely?
          Hide
          Kannan Muthukkaruppan added a comment -

          Stack: Good point on the META keys. A "." or "-" would work better. Those sort lower than a "9" as well.

          Re: timestamp+jenkins_hash--- my feeling is that while we are in the process of modifying the format, might as well go with something stronger like md5 or sha1. Those would be more bits that TS+jenkins hash.

          Show
          Kannan Muthukkaruppan added a comment - Stack: Good point on the META keys. A "." or "-" would work better. Those sort lower than a "9" as well. Re: timestamp+jenkins_hash--- my feeling is that while we are in the process of modifying the format, might as well go with something stronger like md5 or sha1. Those would be more bits that TS+jenkins hash.
          Hide
          stack added a comment -

          .bq Re: timestamp+jenkins_hash--- my feeling is that while we are in the process of modifying the format, might as well go with something stronger like md5 or sha1. Those would be more bits that TS+jenkins hash.

          Better to be safe, agreed.

          Show
          stack added a comment - .bq Re: timestamp+jenkins_hash--- my feeling is that while we are in the process of modifying the format, might as well go with something stronger like md5 or sha1. Those would be more bits that TS+jenkins hash. Better to be safe, agreed.
          Hide
          Kannan Muthukkaruppan added a comment -

          will post a prelim patch on this soon on review board. Unit tests pass. Did some cluster testing, and things seemed fine. Migration tests from older version of HBase servers needs to be done.

          Show
          Kannan Muthukkaruppan added a comment - will post a prelim patch on this soon on review board. Unit tests pass. Did some cluster testing, and things seemed fine. Migration tests from older version of HBase servers needs to be done.
          Show
          Kannan Muthukkaruppan added a comment - http://review.hbase.org/r/104/
          Hide
          HBase Review Board added a comment -

          Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/104/
          -----------------------------------------------------------

          Review request for hbase.

          Summary
          -------

          The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem.

          New region name format:

          <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/

          where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp>

          The old region name format remains:
          <tablename>,<startkey>,<regionIdTimestamp>

          For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form).

          *NOTE*

          ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format.

          In the logs & web ui, old format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>)
          New format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/

          This addresses bug HBASE-2531.

          Diffs


          trunk/bin/add_table.rb 949322
          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322
          trunk/src/main/resources/hbase-webapps/master/table.jsp 949322
          trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322
          trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322

          Diff: http://review.hbase.org/r/104/diff

          Testing
          -------

          unit tests pass. ran some

          Thanks,

          Kannan

          Show
          HBase Review Board added a comment - Message from: "Kannan Muthukkaruppan" <kannan@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/104/ ----------------------------------------------------------- Review request for hbase. Summary ------- The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem. New region name format: <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/ where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp> The old region name format remains: <tablename>,<startkey>,<regionIdTimestamp> For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form). * NOTE * ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format. In the logs & web ui, old format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>) New format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/ This addresses bug HBASE-2531 . Diffs trunk/bin/add_table.rb 949322 trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322 trunk/src/main/resources/hbase-webapps/master/table.jsp 949322 trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322 trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322 Diff: http://review.hbase.org/r/104/diff Testing ------- unit tests pass. ran some Thanks, Kannan
          Hide
          HBase Review Board added a comment -

          Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/104/
          -----------------------------------------------------------

          (Updated 2010-05-28 18:03:20.789555)

          Review request for hbase.

          Summary
          -------

          The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem.

          New region name format:

          <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/

          where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp>

          The old region name format remains:
          <tablename>,<startkey>,<regionIdTimestamp>

          For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form).

          *NOTE*

          ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format.

          In the logs & web ui, old format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>)
          New format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/

          This addresses bug HBASE-2531.

          Diffs


          trunk/bin/add_table.rb 949322
          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322
          trunk/src/main/resources/hbase-webapps/master/table.jsp 949322
          trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322
          trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322

          Diff: http://review.hbase.org/r/104/diff

          Testing (updated)
          -------

          unit tests pass. ran some cluster tests, and things seemed to work ok. Yet to try some migration test (upgrading from an older server).

          Thanks,

          Kannan

          Show
          HBase Review Board added a comment - Message from: "Kannan Muthukkaruppan" <kannan@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/104/ ----------------------------------------------------------- (Updated 2010-05-28 18:03:20.789555) Review request for hbase. Summary ------- The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem. New region name format: <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/ where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp> The old region name format remains: <tablename>,<startkey>,<regionIdTimestamp> For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form). * NOTE * ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format. In the logs & web ui, old format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>) New format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/ This addresses bug HBASE-2531 . Diffs trunk/bin/add_table.rb 949322 trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322 trunk/src/main/resources/hbase-webapps/master/table.jsp 949322 trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322 trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322 Diff: http://review.hbase.org/r/104/diff Testing (updated) ------- unit tests pass. ran some cluster tests, and things seemed to work ok. Yet to try some migration test (upgrading from an older server). Thanks, Kannan
          Hide
          HBase Review Board added a comment -

          Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/104/
          -----------------------------------------------------------

          (Updated 2010-05-28 18:50:19.120776)

          Review request for hbase.

          Changes
          -------

          Forgot to include one new file (hbase/util/MD5Hash.java).

          BTW, also manually ran a test with a cluster created using an older HBase – with regions in old format. Then restarted cluster with new HBase. Newly created regions were in new format as expected. And over time, as old regions, split they migrated over to the new format.

          Summary
          -------

          The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem.

          New region name format:

          <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/

          where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp>

          The old region name format remains:
          <tablename>,<startkey>,<regionIdTimestamp>

          For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form).

          *NOTE*

          ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format.

          In the logs & web ui, old format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>)
          New format region names will show up as:
          <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/

          This addresses bug HBASE-2531.

          Diffs (updated)


          trunk/bin/add_table.rb 949322
          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322
          trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java PRE-CREATION
          trunk/src/main/resources/hbase-webapps/master/table.jsp 949322
          trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322
          trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322
          trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322

          Diff: http://review.hbase.org/r/104/diff

          Testing
          -------

          unit tests pass. ran some cluster tests, and things seemed to work ok. Yet to try some migration test (upgrading from an older server).

          Thanks,

          Kannan

          Show
          HBase Review Board added a comment - Message from: "Kannan Muthukkaruppan" <kannan@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/104/ ----------------------------------------------------------- (Updated 2010-05-28 18:50:19.120776) Review request for hbase. Changes ------- Forgot to include one new file (hbase/util/MD5Hash.java). BTW, also manually ran a test with a cluster created using an older HBase – with regions in old format. Then restarted cluster with new HBase. Newly created regions were in new format as expected. And over time, as old regions, split they migrated over to the new format. Summary ------- The new format for a region name contains its encodedName. The encoded name also serves as the directory name for the region in the filesystem. New region name format: <tablename>,<startkey>,<regionIdTimestamp>/<encodedName>/ where, <encodedName> is a hex version of the MD5 hash of <tablename>,<startkey>,<regionIdTimestamp> The old region name format remains: <tablename>,<startkey>,<regionIdTimestamp> For region names in the old format, the encoded name is a 32-bit JenkinsHash integer value (in its decimal notation, string form). * NOTE * ROOT, the first META region, and regions created by an older version of HBase (0.20 or prior) will continue to use the old region name format. In the logs & web ui, old format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>(<jenkinshashEncodedName>) New format region names will show up as: <tablename>,<startkey>,<regionIdTimestamp>/<md5hashEncodedName>/ This addresses bug HBASE-2531 . Diffs (updated) trunk/bin/add_table.rb 949322 trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 949322 trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java PRE-CREATION trunk/src/main/resources/hbase-webapps/master/table.jsp 949322 trunk/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 949322 trunk/src/test/java/org/apache/hadoop/hbase/TestEmptyMetaInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 949322 trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 949322 Diff: http://review.hbase.org/r/104/diff Testing ------- unit tests pass. ran some cluster tests, and things seemed to work ok. Yet to try some migration test (upgrading from an older server). Thanks, Kannan
          Hide
          HBase Review Board added a comment -

          Message from: stack@duboce.net

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/104/#review100
          -----------------------------------------------------------

          It looks great to me. Only objection is '/' as delimiter in new format. Would prefer something doesn't require escaping when searching in tools such as vim. How about a '.'? I'm going to try it out now. Will report back.

          trunk/bin/add_table.rb
          <http://review.hbase.org/r/104/#comment585>

          I'm sorry you had to edit this file Kannan

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment586>

          Are you tied to '/'? Could we use '.' instead? I'm in vi and having to search for instances of a regionname. I'll have to escape the / before I begin my search? If it was a '.' or '-' I wouldn't have to?

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment587>

          This will come out funny in javadoc I think. You might have to change the '<' into < for them to show properly.

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment588>

          Nice. It'll be easy to change then.

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment589>

          Nitpick: This could be written as:

          return (regionName.length >= 1) && (regionName[regionName.length - 1] == ENC_SEPARATOR);

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment591>

          good

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment592>

          Whats happening here? Why not just return the cached String? Does it not include the new encoded suffix?

          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
          <http://review.hbase.org/r/104/#comment593>

          What changed on this line (and the one before). Just watching out for you. Any changes in hfile cause Ryan to perk up and come looksee.

          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
          <http://review.hbase.org/r/104/#comment594>

          Be careful. Looks only one real change in this file (though the diff reports many more than that)

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          <http://review.hbase.org/r/104/#comment595>

          This is great - getting rid of having to tag on the encoding

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <http://review.hbase.org/r/104/#comment596>

          What changed on this line?

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <http://review.hbase.org/r/104/#comment597>

          What changed on these lines?

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <http://review.hbase.org/r/104/#comment598>

          Yeah, what changed here?

          trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java
          <http://review.hbase.org/r/104/#comment599>

          Throw a RuntimeException I'd say.

          trunk/src/main/resources/hbase-webapps/master/table.jsp
          <http://review.hbase.org/r/104/#comment600>

          This is good

          • stack
          Show
          HBase Review Board added a comment - Message from: stack@duboce.net ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/104/#review100 ----------------------------------------------------------- It looks great to me. Only objection is '/' as delimiter in new format. Would prefer something doesn't require escaping when searching in tools such as vim. How about a '.'? I'm going to try it out now. Will report back. trunk/bin/add_table.rb < http://review.hbase.org/r/104/#comment585 > I'm sorry you had to edit this file Kannan trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment586 > Are you tied to '/'? Could we use '.' instead? I'm in vi and having to search for instances of a regionname. I'll have to escape the / before I begin my search? If it was a '.' or '-' I wouldn't have to? trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment587 > This will come out funny in javadoc I think. You might have to change the '<' into < for them to show properly. trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment588 > Nice. It'll be easy to change then. trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment589 > Nitpick: This could be written as: return (regionName.length >= 1) && (regionName [regionName.length - 1] == ENC_SEPARATOR); trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment591 > good trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment592 > Whats happening here? Why not just return the cached String? Does it not include the new encoded suffix? trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java < http://review.hbase.org/r/104/#comment593 > What changed on this line (and the one before). Just watching out for you. Any changes in hfile cause Ryan to perk up and come looksee. trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java < http://review.hbase.org/r/104/#comment594 > Be careful. Looks only one real change in this file (though the diff reports many more than that) trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java < http://review.hbase.org/r/104/#comment595 > This is great - getting rid of having to tag on the encoding trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < http://review.hbase.org/r/104/#comment596 > What changed on this line? trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < http://review.hbase.org/r/104/#comment597 > What changed on these lines? trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < http://review.hbase.org/r/104/#comment598 > Yeah, what changed here? trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java < http://review.hbase.org/r/104/#comment599 > Throw a RuntimeException I'd say. trunk/src/main/resources/hbase-webapps/master/table.jsp < http://review.hbase.org/r/104/#comment600 > This is good stack
          Hide
          HBase Review Board added a comment -

          Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          http://review.hbase.org/r/104/#review102
          -----------------------------------------------------------

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment601>

          will do.

          trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
          <http://review.hbase.org/r/104/#comment602>

          old style region names don't have their encoded name in the regionNameStr. So I check for that here, and append the encoded name for those regions so that the logs, web UI, etc. will display the encoded named for those regions.

          trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
          <http://review.hbase.org/r/104/#comment603>

          will get rid of all whitespace diffs.

          trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          <http://review.hbase.org/r/104/#comment604>

          Will get rid of all the whitespace diffs. After the big whitespace cleanup landed in 0.20, I decided to set my eclipse to kill trailing whitespaces. But looks like trunk still has a bunch of whitespaces.

          trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java
          <http://review.hbase.org/r/104/#comment605>

          ok...

          • Kannan
          Show
          HBase Review Board added a comment - Message from: "Kannan Muthukkaruppan" <kannan@facebook.com> ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: http://review.hbase.org/r/104/#review102 ----------------------------------------------------------- trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment601 > will do. trunk/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java < http://review.hbase.org/r/104/#comment602 > old style region names don't have their encoded name in the regionNameStr. So I check for that here, and append the encoded name for those regions so that the logs, web UI, etc. will display the encoded named for those regions. trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java < http://review.hbase.org/r/104/#comment603 > will get rid of all whitespace diffs. trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java < http://review.hbase.org/r/104/#comment604 > Will get rid of all the whitespace diffs. After the big whitespace cleanup landed in 0.20, I decided to set my eclipse to kill trailing whitespaces. But looks like trunk still has a bunch of whitespaces. trunk/src/main/java/org/apache/hadoop/hbase/util/MD5Hash.java < http://review.hbase.org/r/104/#comment605 > ok... Kannan
          Hide
          HBase Review Board added a comment -

          Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>

          Show
          HBase Review Board added a comment - Message from: "Kannan Muthukkaruppan" <kannan@facebook.com>
          Hide
          stack added a comment -

          I tried it. I saw stuff like this:

          2010-05-31 11:19:10,258 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS: TestTable,0010151443,1275329539809(717050107): Daughters; TestTable,0010151443,1275329944907/80f60878ba8994c3458931a127d77377/, TestTable,0010376061,1275329944907/9058243430462cd436fc06b748c0aaca/ from sv2borg187,60020,1275329765174; 1 of 1
          

          That looks really good as does a scan of .META. (I can see parents and daughters.... we should get rid of the new-style encoded field in HRegionInfo, but we can do that later).

          Since there are so few things to change – and changing the separator I can do – you want me to fix the above white-spacing, etc., issues on commit Kannan; IOW, you don't have to make a new patch?

          Looks great Kannan.

          Show
          stack added a comment - I tried it. I saw stuff like this: 2010-05-31 11:19:10,258 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_SPLIT_INCLUDES_DAUGHTERS: TestTable,0010151443,1275329539809(717050107): Daughters; TestTable,0010151443,1275329944907/80f60878ba8994c3458931a127d77377/, TestTable,0010376061,1275329944907/9058243430462cd436fc06b748c0aaca/ from sv2borg187,60020,1275329765174; 1 of 1 That looks really good as does a scan of .META. (I can see parents and daughters.... we should get rid of the new-style encoded field in HRegionInfo, but we can do that later). Since there are so few things to change – and changing the separator I can do – you want me to fix the above white-spacing, etc., issues on commit Kannan; IOW, you don't have to make a new patch? Looks great Kannan.
          Hide
          stack added a comment -

          oh, what I did was load an hbase and then I changed the jar to have your patch and restarted... then started loading the table again to prove the patch worked where there was a mix of old and new style encodings. The table seems healthy after letting some splits go on and I don't see exceptions, etc.

          Show
          stack added a comment - oh, what I did was load an hbase and then I changed the jar to have your patch and restarted... then started loading the table again to prove the patch worked where there was a mix of old and new style encodings. The table seems healthy after letting some splits go on and I don't see exceptions, etc.
          Hide
          Kannan Muthukkaruppan added a comment -

          Sure-- feel free to update and commit the patch. Thanks Stack.

          Show
          Kannan Muthukkaruppan added a comment - Sure-- feel free to update and commit the patch. Thanks Stack.
          Hide
          stack added a comment -

          Committed. Thanks for sweet patch Kannan.

          Show
          stack added a comment - Committed. Thanks for sweet patch Kannan.
          Hide
          Todd Lipcon added a comment -

          Why aren't we also changing ROOT and the first META? I'm afraid this will come back to bite us when we actually try to fix meta splitting.

          Show
          Todd Lipcon added a comment - Why aren't we also changing ROOT and the first META? I'm afraid this will come back to bite us when we actually try to fix meta splitting.
          Hide
          stack added a comment -

          .bq Why aren't we also changing ROOT and the first META? I'm afraid this will come back to bite us when we actually try to fix meta splitting.

          I opened a new issue, HBASE-2639, to bring these over too.

          Show
          stack added a comment - .bq Why aren't we also changing ROOT and the first META? I'm afraid this will come back to bite us when we actually try to fix meta splitting. I opened a new issue, HBASE-2639 , to bring these over too.
          Hide
          Kannan Muthukkaruppan added a comment -

          Todd/Stack: ROOT and the first META region are handled in a special hard-coded way in code. See HRegionInfo::ROOT_REGIONINFO and HRegion::FIRST_META_REGIONINFO.

          For example, their region id is hardcoded to 0 & 1 (and they do not use timestamps).

          ROOT,,0
          .META.,,1

          Their Jenkins hash encoded names: 70236052 & 1028785192 respectively do not collide either. Not changing this simplifies handling upgrade from an older version of HBase.

          This should not be an issue for splits of META. If META splits, it should use the new format region names (with timestamps for the region id portion) and md5 hash for encoded names.

          Show
          Kannan Muthukkaruppan added a comment - Todd/Stack: ROOT and the first META region are handled in a special hard-coded way in code. See HRegionInfo::ROOT_REGIONINFO and HRegion::FIRST_META_REGIONINFO. For example, their region id is hardcoded to 0 & 1 (and they do not use timestamps). ROOT ,,0 .META.,,1 Their Jenkins hash encoded names: 70236052 & 1028785192 respectively do not collide either. Not changing this simplifies handling upgrade from an older version of HBase. This should not be an issue for splits of META. If META splits, it should use the new format region names (with timestamps for the region id portion) and md5 hash for encoded names.
          Hide
          stack added a comment -

          @Kannan I moved discussion of root and meta over to hbase-2639. I'll comment there.

          Show
          stack added a comment - @Kannan I moved discussion of root and meta over to hbase-2639. I'll comment there.

            People

            • Assignee:
              Kannan Muthukkaruppan
              Reporter:
              stack
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development