Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8015

Support for Namespaces

        Details

        • Type: New Feature
        • Status: Open
        • Priority: Major
        • Resolution: Unresolved
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: None
        • Labels:
          None
        • Tags:
          0.96notable
        1. Text File
          HBASE-8015_draft_94.patch
          108 kB
          Francis Liu
        2. PDF File
          Namespace Design.pdf
          541 kB
          Francis Liu
        1.
        		 Implement namespace Sub-task Closed Francis Liu
         
        2.
        		 Security support for namespaces Sub-task Closed Vandana Ayyalasomayajula
         
        3.
        		 Basic quota support for namespaces Sub-task Resolved Vandana Ayyalasomayajula
         
        4. Implement rename table tool for namespace migration Sub-task Open Unassigned
         
        5.
        		 Document namespaces in HBase book Sub-task Closed Francis Liu
         
        6.
        		 _acl_ table should be migrated to system namespace Sub-task Resolved Unassigned
         
        7.
        		 Drop the '.' prefix from names under hbase.rootdir after namespaces goes in Sub-task Closed stack
         
        8.
        		 Invert the name of namespace commands to match the snapshots and replication form Sub-task Closed stack
         
        9.
        		 Run NamespaceUpgrade from hbase migration script Sub-task Resolved stack
         
        10.
        		 Mechanism to limit system table deletion and creation in non-secure mode Sub-task Resolved Unassigned
         
        11. Add wal and OpenRegionhandler tier for system namespace Sub-task Open Unassigned
         
        12.
        		 javadoc cleanup of to reflect .META. rename to hbase:meta Sub-task Closed Misty Stanley-Jones
         
        13. Add priority for system tables Sub-task Open Unassigned
         
        14. Adapt the web UIs to namespaces Sub-task Open Unassigned
         
        15. namespace permissions Sub-task Open Unassigned
         
        16.
        		 create_namespace with property value throws error Sub-task Closed Enis Soztutar
         
        17.
        		 namespace permissions - add support from table creation privilege in a namespace 'C' Sub-task Closed Huaiyu Zhu
         
        18. namespace permissions - add support for namespace list privilege 'L' Sub-task Open Unassigned
         
        19.
        		 Quota support for namespace should take region merge into account Sub-task Resolved Ted Yu
         
        20.
        		 Quota support for namespace should take snapshot restore and clone into account Sub-task Resolved Ashish Singhi
         
        21.
        		 REST Support for Namespaces Sub-task Resolved Matt Warhaftig
         
        22. Document REST endpoints for namespaces Sub-task Open Unassigned
         

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Initial draft of design. This originally was intended to be implemented as coprocessors thus it's design was made to be as non-invasive as possible.

          Enis Soztutar Suggested that it would be better to make this part of core. I'd be up for doing that and open to other changes to make things more integrated.

          Show
          toffer Francis Liu added a comment - Initial draft of design. This originally was intended to be implemented as coprocessors thus it's design was made to be as non-invasive as possible. Enis Soztutar Suggested that it would be better to make this part of core. I'd be up for doing that and open to other changes to make things more integrated.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          +1 on making core (as opposed to cp-s I assume?)
          "Given the level of autonomy namespaces will provide tenants. " what does this mean?
          What kind of resource allocation will be possible, except for server groups (just examples, to understand it better)?

          Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.
          Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?
          Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

          Show
          sershe Sergey Shelukhin added a comment - +1 on making core (as opposed to cp-s I assume?) "Given the level of autonomy namespaces will provide tenants. " what does this mean? What kind of resource allocation will be possible, except for server groups (just examples, to understand it better)? Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat. Also, how do we handle existing backward compat if someone already has a table name "foo.bar"? Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          +1 on making core (as opposed to cp-s I assume?)

          Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

          "Given the level of autonomy namespaces will provide tenants. " what does this mean?

          In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

          {quota}
          Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.{quota}

          Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

          {quota}
          Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?{quota}

          Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

          {quota}
          Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?{quota}

          As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

          Show
          toffer Francis Liu added a comment - +1 on making core (as opposed to cp-s I assume?) Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality. "Given the level of autonomy namespaces will provide tenants. " what does this mean? In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves. {quota} Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.{quota} Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this {quota} Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?{quota} Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used? {quota} Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?{quota} As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          +1 on having namespaces in core. Namespace/database's are universally understood in terms of the database space. We can keep the grouping of regionservers per namespace out of core, and deliver that as a part of the other region grouping issue.

          Show
          enis Enis Soztutar added a comment - +1 on having namespaces in core. Namespace/database's are universally understood in terms of the database space. We can keep the grouping of regionservers per namespace out of core, and deliver that as a part of the other region grouping issue.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          w.r.t. dot in table name:

            public static final String VALID_USER_TABLE_REGEX = "(?:[a-zA-Z_0-9][a-zA-Z_0-9.-]*)";

          So dot should be allowed.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - w.r.t. dot in table name: public static final String VALID_USER_TABLE_REGEX = "(?:[a-zA-Z_0-9][a-zA-Z_0-9.-]*)" ; So dot should be allowed.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          reposting coz of typos:

          +1 on making core (as opposed to cp-s I assume?)

          Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

          "Given the level of autonomy namespaces will provide tenants. " what does this mean?

          In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

          Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.

          Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

          Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?

          Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

          Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

          As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

          Show
          toffer Francis Liu added a comment - reposting coz of typos: +1 on making core (as opposed to cp-s I assume?) Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality. "Given the level of autonomy namespaces will provide tenants. " what does this mean? In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves. Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat. Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this Also, how do we handle existing backward compat if someone already has a table name "foo.bar"? Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used? Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes? As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          @Francis:
          In the future when you want to correct your previous comment, you can quote the previous one and provide correction below the quote.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Francis: In the future when you want to correct your previous comment, you can quote the previous one and provide correction below the quote.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          HBASE-7999 provides System table support.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - HBASE-7999 provides System table support.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          w.r.t. dot in table name:

          public static final String VALID_USER_TABLE_REGEX = "(?:[a-zA-Z_0-9][a-zA-Z_0-9.-]*)";

          So dot should be allowed

          Thanks for the clarification. It seems like '.' and '-' isn't allowed only if it's the first character. For backward compatibility why don't we create namespaces for those tables that are named that way?

          Show
          toffer Francis Liu added a comment - w.r.t. dot in table name: public static final String VALID_USER_TABLE_REGEX = "(?: [a-zA-Z_0-9] [a-zA-Z_0-9.-] *)"; So dot should be allowed Thanks for the clarification. It seems like '.' and '-' isn't allowed only if it's the first character. For backward compatibility why don't we create namespaces for those tables that are named that way?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

          I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

          Show
          enis Enis Soztutar added a comment - Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

          They will be first class citizens. There will be a namespace table for namespace meta information. Also embedding namespace information in table name does not prevent this, we are just introducing the notion of fully-qualified table names (which we should introduce anyway). And store them in this form in meta/root.

          Show
          toffer Francis Liu added a comment - I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc. They will be first class citizens. There will be a namespace table for namespace meta information. Also embedding namespace information in table name does not prevent this, we are just introducing the notion of fully-qualified table names (which we should introduce anyway). And store them in this form in meta/root.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Given that '.' is already used. Should we pick another delimiter for namespaces? Or should we provide a backward compatible way to support this. Ie creating namespaces for tablenames with '.'?

          Show
          toffer Francis Liu added a comment - Given that '.' is already used. Should we pick another delimiter for namespaces? Or should we provide a backward compatible way to support this. Ie creating namespaces for tablenames with '.'?
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          The latter may not necessarily be backward compatible...

          Show
          sershe Sergey Shelukhin added a comment - The latter may not necessarily be backward compatible...
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Good point. For now I'll stick with '/' if there are no objections. And revisit '.' if anyone feels strongly about it.

          Show
          toffer Francis Liu added a comment - Good point. For now I'll stick with '/' if there are no objections. And revisit '.' if anyone feels strongly about it.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Hmm, ns.table is definitely more intuitive than /. SQL also uses that convention.

          Show
          enis Enis Soztutar added a comment - Hmm, ns.table is definitely more intuitive than /. SQL also uses that convention.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          my point about backward compat was about the storage, not display. I.e. if people want to create a table "a.b" without namespace and be confused while viewing some table list, I think it's ok. Maybe we can disallow tables with dots.
          If they have existing table "a.b" and NS is stored separately, again the worst thing that happens is user gets confused, reads the release notes and renames it. Should be ok to.
          But if we /store/ NS in table name, then with table a.b, without special handing, system gets confused (thinks it's in NS "a") after upgrade

          Show
          sershe Sergey Shelukhin added a comment - my point about backward compat was about the storage, not display. I.e. if people want to create a table "a.b" without namespace and be confused while viewing some table list, I think it's ok. Maybe we can disallow tables with dots. If they have existing table "a.b" and NS is stored separately, again the worst thing that happens is user gets confused, reads the release notes and renames it. Should be ok to. But if we /store/ NS in table name, then with table a.b, without special handing, system gets confused (thinks it's in NS "a") after upgrade
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I see, I am assuming we have to store namespace as part of the region name. And store them fully-qualified on hdfs/zookeeper/etc. Else we would be forced to have all table names to be globally unique which would different from database semantics.

          Another concern is if a user can run the same application code against 0.96.

          ie if I wanted to scan:

          > scan 'foo.bar'

          Pre-NS this would scan table 'foo.bar'. Post-NS, the system would parse this out as table bar in namespace foo. One way we could deal with this is read the table as Post-NS if it doesn't access read it as Pre-NS, check again if not then fail.

          Show
          toffer Francis Liu added a comment - I see, I am assuming we have to store namespace as part of the region name. And store them fully-qualified on hdfs/zookeeper/etc. Else we would be forced to have all table names to be globally unique which would different from database semantics. Another concern is if a user can run the same application code against 0.96. ie if I wanted to scan: > scan 'foo.bar' Pre-NS this would scan table 'foo.bar'. Post-NS, the system would parse this out as table bar in namespace foo. One way we could deal with this is read the table as Post-NS if it doesn't access read it as Pre-NS, check again if not then fail.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          The latter may not necessarily be backward compatible...

          Can you give an example to this? Thinking about it more, it seems to me you'll just end up with a lot of namespaces?

          Show
          toffer Francis Liu added a comment - The latter may not necessarily be backward compatible... Can you give an example to this? Thinking about it more, it seems to me you'll just end up with a lot of namespaces?
          Hide
          Permalink
          jesse_yates Jesse Yates added a comment - - edited

          my point about backward compat was about the storage

          IIRC we don't support backward compat for 0.96 backwards. As long as this goes into trunk, it should be fine going forward.

          Show
          jesse_yates Jesse Yates added a comment - - edited my point about backward compat was about the storage IIRC we don't support backward compat for 0.96 backwards. As long as this goes into trunk, it should be fine going forward.
          Hide
          Permalink
          jesse_yates Jesse Yates added a comment -

          Francis Liu what kind of timeline are you planning for doing this work? I love to see it subsume HBASE-7999, but this is a much more involved change than what's going on there.

          Show
          jesse_yates Jesse Yates added a comment - Francis Liu what kind of timeline are you planning for doing this work? I love to see it subsume HBASE-7999 , but this is a much more involved change than what's going on there.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          Well, there will be an upgrade script... then this will have to be added to it.
          Also, trunk is now 98 so technically this should go into 95 then

          Show
          sershe Sergey Shelukhin added a comment - Well, there will be an upgrade script... then this will have to be added to it. Also, trunk is now 98 so technically this should go into 95 then
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          HBASE-7999 has 0.96 as Fix Version.
          If we agree that this feature is superset of HBASE-7999, we should focus on this feature.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - HBASE-7999 has 0.96 as Fix Version. If we agree that this feature is superset of HBASE-7999 , we should focus on this feature.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Jesse Yates I'm hoping to get an dev drop (0.94) by end of next week. I'll try to squeeze in a trunk patch. Does that work for you?

          Show
          toffer Francis Liu added a comment - Jesse Yates I'm hoping to get an dev drop (0.94) by end of next week. I'll try to squeeze in a trunk patch. Does that work for you?
          Hide
          Permalink
          jesse_yates Jesse Yates added a comment -

          Yeah, that would be great! Thanks Francis

          Show
          jesse_yates Jesse Yates added a comment - Yeah, that would be great! Thanks Francis
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          @Francis:
          Can you work on the trunk patch first ?
          If it gets ready before 0.95 RC0, it can go into 0.95

          Thanks

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Francis: Can you work on the trunk patch first ? If it gets ready before 0.95 RC0, it can go into 0.95 Thanks
          Hide
          Permalink
          toffer Francis Liu added a comment -

          So it seems we're fine providing a migration script. So I've gone ahead and produced a rough patch with the change.

          The general idea is to have a TableName class which encapsulates both namespace and tableQualifier (AKA the old table name). And this will be what gets passed around as much as possible internally. Table names are referenced fully qualified otherwise.
          I've also changed the tableDir on hdfs to include namespace as the parent directory. This will enable us to set HDFS quotas by namespace.

          I skipped the CRUD related stuff since that seems pretty straightforward. Let me know if this approach is acceptable.

          Show
          toffer Francis Liu added a comment - So it seems we're fine providing a migration script. So I've gone ahead and produced a rough patch with the change. The general idea is to have a TableName class which encapsulates both namespace and tableQualifier (AKA the old table name). And this will be what gets passed around as much as possible internally. Table names are referenced fully qualified otherwise. I've also changed the tableDir on hdfs to include namespace as the parent directory. This will enable us to set HDFS quotas by namespace. I skipped the CRUD related stuff since that seems pretty straightforward. Let me know if this approach is acceptable.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          I've also changed the tableDir on hdfs to include namespace as the parent directory

          How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

          Show
          enis Enis Soztutar added a comment - I've also changed the tableDir on hdfs to include namespace as the parent directory How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible.

          Maybe we should have WALs per namespace per server . If we want other namespace based features (ie replication) going this route makes sense?

          Internally we plan to do it using regionserver groups (one namespace per group). Since we don't want our tenants contending for other resources as well (disk, mem, cpu, etc).

          Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

          It'd be good if we can leverage hdfs quota since it's already there and has worked well (at least for us). That's also less complexity for us, then we can focus on adding HBase specific Quota (ie regions, tables, etc).

          Show
          toffer Francis Liu added a comment - How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Maybe we should have WALs per namespace per server . If we want other namespace based features (ie replication) going this route makes sense? Internally we plan to do it using regionserver groups (one namespace per group). Since we don't want our tenants contending for other resources as well (disk, mem, cpu, etc). Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons. It'd be good if we can leverage hdfs quota since it's already there and has worked well (at least for us). That's also less complexity for us, then we can focus on adding HBase specific Quota (ie regions, tables, etc).
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Some questions on design doc.

          multiple namespaces can be part of the same region server group.

          Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
          I find the latter more intuitive.

          hbase ­ admin tables:­ROOT­, .META., ACLs

          Would the group table from HBASE­6721 belong to the above namespace ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Some questions on design doc. multiple namespaces can be part of the same region server group. Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group). I find the latter more intuitive. hbase ­ admin tables:­ROOT­, .META., ACLs Would the group table from HBASE­6721 belong to the above namespace ?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Sorry for the delay I was OOO last week. The patch is way bigger than I anticipated changing the path hierarchy took bulk of it as assumptions about it was all over the code. Here's an initial draft of the patch, functionally it's complete. I need some feedback before I can go further.

          https://reviews.apache.org/r/10167/diff/#index_header

          Show
          toffer Francis Liu added a comment - Sorry for the delay I was OOO last week. The patch is way bigger than I anticipated changing the path hierarchy took bulk of it as assumptions about it was all over the code. Here's an initial draft of the patch, functionally it's complete. I need some feedback before I can go further. https://reviews.apache.org/r/10167/diff/#index_header
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
          I find the latter more intuitive.

          If you find that agreeable. We can keep it that way.

          Would the group table from HBASE­6721 belong to the above namespace ?

          Yep. depending on which ends up going in first will have to make that change.

          Show
          toffer Francis Liu added a comment - Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group). I find the latter more intuitive. If you find that agreeable. We can keep it that way. Would the group table from HBASE­6721 belong to the above namespace ? Yep. depending on which ends up going in first will have to make that change.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          I am beginning to go over the code on review board.

          -there's a special case with tables in the default namespace: <table qualifier>

          What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.

          I've renamed the meta table to become -hbase.meta

          Nice. This is the same as what was proposed over in HBASE-8093

          Show
          yuzhihong@gmail.com Ted Yu added a comment - I am beginning to go over the code on review board. -there's a special case with tables in the default namespace: <table qualifier> What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case. I've renamed the meta table to become -hbase . meta Nice. This is the same as what was proposed over in HBASE-8093
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Looks like the latest patch wasn't properly generated.
          In HMaster.java:

          +<<<<<<< HEAD
+=======
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ServerCallable;
+import org.apache.hadoop.hbase.exceptions.ConstraintException;
+import org.apache.hadoop.hbase.exceptions.DeserializationException;
+>>>>>>> first forward port of namespace trunk patch
          Show
          yuzhihong@gmail.com Ted Yu added a comment - Looks like the latest patch wasn't properly generated. In HMaster.java: +<<<<<<< HEAD +======= + import org.apache.hadoop.hbase.NamespaceDescriptor; + import org.apache.hadoop.hbase.TableName; + import org.apache.hadoop.hbase.client.ServerCallable; + import org.apache.hadoop.hbase.exceptions.ConstraintException; + import org.apache.hadoop.hbase.exceptions.DeserializationException; +>>>>>>> first forward port of namespace trunk patch
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Thanks for reviewing ted.

          What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.

          It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. They will all fall into the default namespace bucket and be accessed by the application the same way.

          Show
          toffer Francis Liu added a comment - Thanks for reviewing ted. What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case. It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. They will all fall into the default namespace bucket and be accessed by the application the same way.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          This of default namespace akin to the default package in java.

          Show
          toffer Francis Liu added a comment - This of default namespace akin to the default package in java.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Looks like the latest patch wasn't properly generated.
          In HMaster.java:

          Sorry about that. Updated the patch.

          Show
          toffer Francis Liu added a comment - Looks like the latest patch wasn't properly generated. In HMaster.java: Sorry about that. Updated the patch.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          I've renamed the meta table to become -hbase.meta

          Nice. This is the same as what was proposed over in HBASE-8093

          Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

          Show
          enis Enis Soztutar added a comment - I've renamed the meta table to become -hbase.meta Nice. This is the same as what was proposed over in HBASE-8093 Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed.

          I am still going through your code.
          Would user be able to create table in default namespace after migration ?
          Put in another way: what privilege is required to do the above ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. I am still going through your code. Would user be able to create table in default namespace after migration ? Put in another way: what privilege is required to do the above ?
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          TestMasterFailover failed with patch. In test output, I saw:

              <error message="test timed out after 180000 milliseconds" type="java.lang.Exception">java.lang.Exception: test timed out after 180000 milliseconds
  at java.lang.Thread.sleep(Native Method)
  at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:133)
  at org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:469)
  at org.apache.hadoop.hbase.HBaseCluster.waitForActiveAndReadyMaster(HBaseCluster.java:206)
  at org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:759)
...
2013-03-28 16:27:21,666 INFO  [Master:0;10.11.2.252,51376,1364513236949] master.HMaster(954): Failed to contact server
java.io.IOException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
  at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
  at com.sun.proxy.$Proxy17.getRegionInfo(Unknown Source)
  at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRegionInfo(ProtobufUtil.java:1269)
  at org.apache.hadoop.hbase.master.HMaster.assignSystemTables(HMaster.java:949)
  at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:780)
  at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:536)
  at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol
  at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:166)
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1871)

  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1321)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
  ... 6 more
...
2013-03-28 16:30:39,937 INFO  [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.RegionStates(264): Region {NAME => '-hbase-.-meta-,,1', STARTKEY => '', ENDKEY => '', ENCODED => 715583861,} transitioned from {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082} to {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082}
2013-03-28 16:30:39,938 WARN  [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.AssignmentManager(1869): Failed assignment of -hbase-.-meta-,,1.715583861 to 10.11.2.252,51480,1364513407082, trying to assign elsewhere instead; try=10 of 10
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: /10.11.2.252:51480
  at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:785)
  at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1414)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1299)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
  at com.sun.proxy.$Proxy17.openRegion(Unknown Source)
  at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:603)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1796)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1431)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1410)
  at org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(AssignmentManager.java:2303)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:100)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:124)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:69)
  at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
          Show
          yuzhihong@gmail.com Ted Yu added a comment - TestMasterFailover failed with patch. In test output, I saw: <error message= "test timed out after 180000 milliseconds" type= "java.lang.Exception" >java.lang.Exception: test timed out after 180000 milliseconds at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:133) at org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:469) at org.apache.hadoop.hbase.HBaseCluster.waitForActiveAndReadyMaster(HBaseCluster.java:206) at org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:759) ... 2013-03-28 16:27:21,666 INFO [Master:0;10.11.2.252,51376,1364513236949] master.HMaster(954): Failed to contact server java.io.IOException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146) at com.sun.proxy.$Proxy17.getRegionInfo(Unknown Source) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRegionInfo(ProtobufUtil.java:1269) at org.apache.hadoop.hbase.master.HMaster.assignSystemTables(HMaster.java:949) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:780) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:536) at java.lang. Thread .run( Thread .java:680) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:166) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1871) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1321) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131) ... 6 more ... 2013-03-28 16:30:39,937 INFO [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.RegionStates(264): Region {NAME => '-hbase-.-meta-,,1', STARTKEY => '', ENDKEY => '', ENCODED => 715583861,} transitioned from {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082} to {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082} 2013-03-28 16:30:39,938 WARN [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.AssignmentManager(1869): Failed assignment of -hbase-.-meta-,,1.715583861 to 10.11.2.252,51480,1364513407082, trying to assign elsewhere instead; try =10 of 10 org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: /10.11.2.252:51480 at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:785) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1414) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1299) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131) at com.sun.proxy.$Proxy17.openRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:603) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1796) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1431) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1410) at org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(AssignmentManager.java:2303) at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:100) at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:124) at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:69) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I am still going through your code.
          Would user be able to create table in default namespace after migration ?
          Put in another way: what privilege is required to do the above ?

          namespace affiliation is determine by table name during creation.

          ie

          #create table in namespace foo
          create 'foo.my_table','f'

          #create table in namespace default
          create 'my_table','f'

          Show
          toffer Francis Liu added a comment - I am still going through your code. Would user be able to create table in default namespace after migration ? Put in another way: what privilege is required to do the above ? namespace affiliation is determine by table name during creation. ie #create table in namespace foo create 'foo.my_table','f' #create table in namespace default create 'my_table','f'
          Hide
          Permalink
          toffer Francis Liu added a comment -

          TestMasterFailover failed with patch. In test output, I saw:

          Yep, as I mentioned in reviewboard. There are 3 tests that are failing. Have yet to determine if it is related to this patch.

          Show
          toffer Francis Liu added a comment - TestMasterFailover failed with patch. In test output, I saw: Yep, as I mentioned in reviewboard. There are 3 tests that are failing. Have yet to determine if it is related to this patch.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Without the patch, I got:

          Running org.apache.hadoop.hbase.master.TestMasterFailover
          2013-03-28 16:57:21.814 java[4597:db03] Unable to load realm info from SCDynamicStore
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Without the patch, I got: Running org.apache.hadoop.hbase.master.TestMasterFailover 2013-03-28 16:57:21.814 java [4597:db03] Unable to load realm info from SCDynamicStore Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

          AFAIK, if the META table goes offline due to RS failure. There is no special case to make it's assignment a priority. And priority is based on lexical ordering, in which case we will need the funky prefix. Not unless we'd want to add complexity to handle that logic.

          BTW the check is not in yet. But I'd also like to remove '.' as an option for table name. Beyond the proposed namespace delimiter usage.

          Show
          toffer Francis Liu added a comment - Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff. AFAIK, if the META table goes offline due to RS failure. There is no special case to make it's assignment a priority. And priority is based on lexical ordering, in which case we will need the funky prefix. Not unless we'd want to add complexity to handle that logic. BTW the check is not in yet. But I'd also like to remove '.' as an option for table name. Beyond the proposed namespace delimiter usage.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Without the patch, I got:

          Running org.apache.hadoop.hbase.master.TestMasterFailover
          2013-03-28 16:57:21.814 java[4597:db03] Unable to load realm info from SCDynamicStore
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec

          Oh, maybe it's just my machine then. What was the purpose of your previous comment then?

          Show
          toffer Francis Liu added a comment - Without the patch, I got: Running org.apache.hadoop.hbase.master.TestMasterFailover 2013-03-28 16:57:21.814 java [4597:db03] Unable to load realm info from SCDynamicStore Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec Oh, maybe it's just my machine then. What was the purpose of your previous comment then?
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          I plan to revisit this test after finishing one round of review.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - I plan to revisit this test after finishing one round of review.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Francis Liu, I would like to get namespaces in 0.96, because it fits well into the singularity. Getting it in later would be hard. I've put up some comments at RB. I can offer some help should you need with review / split / etc.

          Show
          enis Enis Soztutar added a comment - Francis Liu , I would like to get namespaces in 0.96, because it fits well into the singularity. Getting it in later would be hard. I've put up some comments at RB. I can offer some help should you need with review / split / etc.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Enis Soztutar I would very much like to get it in as well. I've addressed your comments. Also I've broken down the pieces into subtasks. The patch on RB is for the first subtask. We currently have 1-3 running on 0.94. We'll forward port the 2nd and 3rd piece as soon as the first task completed to keep things manageable.

          Show
          toffer Francis Liu added a comment - Enis Soztutar I would very much like to get it in as well. I've addressed your comments. Also I've broken down the pieces into subtasks. The patch on RB is for the first subtask. We currently have 1-3 running on 0.94. We'll forward port the 2nd and 3rd piece as soon as the first task completed to keep things manageable.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I've updated the design doc with a component diagram and some notes to make reviewing the code a bit easier. Let me know.

          Show
          toffer Francis Liu added a comment - I've updated the design doc with a component diagram and some notes to make reviewing the code a bit easier. Let me know.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Thanks Francis, I'll take a look at this today.

          Show
          enis Enis Soztutar added a comment - Thanks Francis, I'll take a look at this today.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Going over the design doc, I have a few questions:

          As a first step we only intend to limit the number of tables and regions a given namespace may contain.

          Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?

          namespace_create ‘<namespace>’, {

          What about naming the new command create_namespace (with verb at the beginning) ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Going over the design doc, I have a few questions: As a first step we only intend to limit the number of tables and regions a given namespace may contain. Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ? namespace_create ‘<namespace>’, { What about naming the new command create_namespace (with verb at the beginning) ?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?

          That's disk space quota. Right now we plan to enforce it using hdfs quota and see how that goes. That way we don't have to reinvent the wheel.

          What about naming the new command create_namespace (with verb at the beginning) ?

          I wanted to namespace the commands . Let me know if the convention is different for HBase.

          Show
          toffer Francis Liu added a comment - Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ? That's disk space quota. Right now we plan to enforce it using hdfs quota and see how that goes. That way we don't have to reinvent the wheel. What about naming the new command create_namespace (with verb at the beginning) ? I wanted to namespace the commands . Let me know if the convention is different for HBase.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?

          For security, should each namespace have its own permission settings ?

          For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ? For security, should each namespace have its own permission settings ? For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          @Francis:
          Can you add new tests for this feature ?
          The new tests should include unit tests and, integration tests.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Francis: Can you add new tests for this feature ? The new tests should include unit tests and, integration tests.
          Hide
          Permalink
          stack stack added a comment -

          Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

          It has been suggested that we make a branch to speed dev on this issue which should help move things along; a branch will need a sponsor/conmiitter. I could volunteer but there are probably better mentors than sloppy I who can help along this issue and advise on what is needed to make the cut-off date fast looming.

          Show
          stack stack added a comment - Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name. It has been suggested that we make a branch to speed dev on this issue which should help move things along; a branch will need a sponsor/conmiitter. I could volunteer but there are probably better mentors than sloppy I who can help along this issue and advise on what is needed to make the cut-off date fast looming.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          I volunteer to be the sponsor.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - I volunteer to be the sponsor.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?

          When I was asking about a different delimiter. No one was really happy with anything else but a ".". If you have a suggestion please let us know.

          Show
          toffer Francis Liu added a comment - Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ? When I was asking about a different delimiter. No one was really happy with anything else but a ".". If you have a suggestion please let us know.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          @Francis:
          Can you add new tests for this feature ?
          The new tests should include unit tests and, integration tests.

          There are unit tests I can have it run as an IT test as well. Though the real test really is the existing tests as the main issues are updating/replacing code segments which make assumptions about the directory layout. The tricky ones were in snapshots, hbck and splits. In this patch I've gotten all of those to pass. We can add more tests to these components as needed.

          Show
          toffer Francis Liu added a comment - @Francis: Can you add new tests for this feature ? The new tests should include unit tests and, integration tests. There are unit tests I can have it run as an IT test as well. Though the real test really is the existing tests as the main issues are updating/replacing code segments which make assumptions about the directory layout. The tricky ones were in snapshots, hbck and splits. In this patch I've gotten all of those to pass. We can add more tests to these components as needed.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          For security, should each namespace have its own permission settings ?

          Yes, the permissions are similar to to directory privileges. You would need write access to a namespace to be able to create and drop tables. We already have a patch for this in 0.94. We'll put it up once we sort out the core patch.

          For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

          yep see subtask. We also have this implemented will put up patch once most things are sorted out with core patch.

          Show
          toffer Francis Liu added a comment - For security, should each namespace have its own permission settings ? Yes, the permissions are similar to to directory privileges. You would need write access to a namespace to be able to create and drop tables. We already have a patch for this in 0.94. We'll put it up once we sort out the core patch. For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ? yep see subtask. We also have this implemented will put up patch once most things are sorted out with core patch.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          @Francis:
          Take a look at the summary here:
          http://search-hadoop.com/m/K0wzj1V5vpu

          Delimiter and directory structure for namespace support is discussed in the middle of the above summary.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - @Francis: Take a look at the summary here: http://search-hadoop.com/m/K0wzj1V5vpu Delimiter and directory structure for namespace support is discussed in the middle of the above summary.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Ted Yu thanks but there doesn't seem to be any consensus?

          Show
          toffer Francis Liu added a comment - Ted Yu thanks but there doesn't seem to be any consensus?
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          That's right.

          Maybe you can start discussion on dev@hbase for this topic.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - That's right. Maybe you can start discussion on dev@hbase for this topic.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

          stack Would we be able to get it in quicker with a feature branch? The reason I am asking is because part of what's challenging about this enhancement is catching all the code which makes assumption on path changes. And that will continue to grow until we provide the helper functions to be used. We have an 0.94 version of this patch working in sandbox with active users. So I believe basic functionality is working. The big thing missing is the migration script (We don't use '.' in table names).

          Show
          toffer Francis Liu added a comment - Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name. stack Would we be able to get it in quicker with a feature branch? The reason I am asking is because part of what's challenging about this enhancement is catching all the code which makes assumption on path changes. And that will continue to grow until we provide the helper functions to be used. We have an 0.94 version of this patch working in sandbox with active users. So I believe basic functionality is working. The big thing missing is the migration script (We don't use '.' in table names).
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Another option is to factor the helper piece out of this class and get that committed quicker. Apart from extra work, we won't be able to verify everything is covered without the rest of the patch. Thoughts?

          Show
          toffer Francis Liu added a comment - Another option is to factor the helper piece out of this class and get that committed quicker. Apart from extra work, we won't be able to verify everything is covered without the rest of the patch. Thoughts?
          Hide
          Permalink
          eclark Elliott Clark added a comment -

          I think that adding a parameter for namespace is cleaner. It would mean that we don't add need to rename tables. People can continue to keep their table names. We also then get an easier path to infer the default namespace.

          Show
          eclark Elliott Clark added a comment - I think that adding a parameter for namespace is cleaner. It would mean that we don't add need to rename tables. People can continue to keep their table names. We also then get an easier path to infer the default namespace.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Elliott Clark We can add that an the interface level to support both means of identifying tables. At the very least we will have to store table names fully qualified in the catalog tables. Storing it on hdfs and zk the same way makes things easier as well.

          By default namespace did you mean something like a "use <namespace>" command in the cli?

          Show
          toffer Francis Liu added a comment - Elliott Clark We can add that an the interface level to support both means of identifying tables. At the very least we will have to store table names fully qualified in the catalog tables. Storing it on hdfs and zk the same way makes things easier as well. By default namespace did you mean something like a "use <namespace>" command in the cli?
          Hide
          Permalink
          eclark Elliott Clark added a comment - - edited

          At the very least we will have to store table names fully qualified in the catalog tables.

          True, but we don't need to use . here, or expose it to the user.
          So I'm saying most places should use a different character to store the table name, and I would argue that we shouldn't expose that fully qualified table name to the user at all.

          With a namespace

          create 'namespace', 't1', 'f1'

          Places something in meta like: 'namespace,t1,,hash'

          Put Usage:

          put 'namespace', 't1', 'r1', 'c1', 'value'

          or

          use 'namespace'
put 't1', 'r1', 'c1', 'value'

          default space

          create 't1', 'f1'

          places something in meta like: 'default,t1,,hash'

          put 't1', 'r1', 'c1', 'value'

          or

          put 'default', 't1', 'r1', 'c1', 'value'

          Comma is already reserved and we can use it for everywhere that we need a string name. But we shouldn't expose that to the user. We shouldn't overload table name with two different parts. It gets confusing for users when one parameter actually represents two different things (namespace and table name).

          Show
          eclark Elliott Clark added a comment - - edited At the very least we will have to store table names fully qualified in the catalog tables. True, but we don't need to use . here, or expose it to the user. So I'm saying most places should use a different character to store the table name, and I would argue that we shouldn't expose that fully qualified table name to the user at all. With a namespace create 'namespace', 't1', 'f1' Places something in meta like: 'namespace,t1,,hash' Put Usage: put 'namespace', 't1', 'r1', 'c1', 'value' or use 'namespace' put 't1', 'r1', 'c1', 'value' default space create 't1', 'f1' places something in meta like: 'default,t1,,hash' put 't1', 'r1', 'c1', 'value' or put ' default ', 't1', 'r1', 'c1', 'value' Comma is already reserved and we can use it for everywhere that we need a string name. But we shouldn't expose that to the user. We shouldn't overload table name with two different parts. It gets confusing for users when one parameter actually represents two different things (namespace and table name).
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Should we support cross-namespace constructs ? If so, namespace would be exposed to user.

          For Oracle, some Namespaces are reserved:
          http://docs.oracle.com/cd/B28359_01/appdev.111/b31231/appb.htm#BABFFDFC

          This means an ordinary user query can easily reference more than one namespace.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Should we support cross-namespace constructs ? If so, namespace would be exposed to user. For Oracle, some Namespaces are reserved: http://docs.oracle.com/cd/B28359_01/appdev.111/b31231/appb.htm#BABFFDFC This means an ordinary user query can easily reference more than one namespace.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I see. We need to expose it mainly for backwards compatibility. Also isn't being able to reference a fully-qualified table name common practice in databases? So users aren't learning something new here.

          Show
          toffer Francis Liu added a comment - I see. We need to expose it mainly for backwards compatibility. Also isn't being able to reference a fully-qualified table name common practice in databases? So users aren't learning something new here.
          Hide
          Permalink
          eclark Elliott Clark added a comment -

          Should we support cross-namespace constructs ? If so, namespace would be exposed to user.

          From my point of view the most important usability concern should be the java api, with thrift second. My proposal would allow cross namespace very easily by just creating two different HTable's in different namespaces.

          The shell is nice but it's just a convenience utility. We shouldn't make the most used case harder or more confusing chasing some dream of having a sql shell. In my opinion since HBase is not Oracle's 11g, it shouldn't value being like their sql.

          We need to expose it mainly for backwards compatibility.

          Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

          Show
          eclark Elliott Clark added a comment - Should we support cross-namespace constructs ? If so, namespace would be exposed to user. From my point of view the most important usability concern should be the java api, with thrift second. My proposal would allow cross namespace very easily by just creating two different HTable's in different namespaces. The shell is nice but it's just a convenience utility. We shouldn't make the most used case harder or more confusing chasing some dream of having a sql shell. In my opinion since HBase is not Oracle's 11g, it shouldn't value being like their sql. We need to expose it mainly for backwards compatibility. Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

          Essentially we need to keep the "HTable(byte[] tableName, Configuration config)" constructor. The current implementation would assume a table is part of the default namespace if only the table name is provided.

          Apart from potentially confusing our users. Do you have any other concerns against supporting a fully-qualified table name?

          Show
          toffer Francis Liu added a comment - Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner). Essentially we need to keep the "HTable(byte[] tableName, Configuration config)" constructor. The current implementation would assume a table is part of the default namespace if only the table name is provided. Apart from potentially confusing our users. Do you have any other concerns against supporting a fully-qualified table name?
          Hide
          Permalink
          stack stack added a comment -

          Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

          IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

          Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

          Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

          If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

          +1 on getting rid of funky system tables stuff (but yeah, as per Francis above, they were given the funny names so they sorted before user tables).

          Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

          On the design, needs date, author, and JIRA number.

          As a first step we only intend to limit the number of tables and regions a given namespace may contain.

          This can be done later, right? You'd have to do it in hbase.

          Are we saying the default ns is called 'default'? Or that it has no name?

          NamespaceController coprocessor running on region servers to correctly enforce quota restrictions

          Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)

          How much of this design has to be in place for 0.96 to ship?

          <ROOT>/data/<NAMESPACE>/<TABLE>

          Why the need for 'data' in above?

          Looking quickly at the attached patch (will look at rb later):

          Below is interesting (smile)

          -        <hadoop.version>0.23.3</hadoop.version>
+        <hadoop.version>0.23.7-SNAPSHOT</hadoop.version>

          Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).

          NamespaceDescriptor implements java Serializable?

          NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

          Will look at rb patch next. What is attached doesn't seem enough. Will be back.

          Show
          stack stack added a comment - Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here) IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this. Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going. Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out) If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above. +1 on getting rid of funky system tables stuff (but yeah, as per Francis above, they were given the funny names so they sorted before user tables). Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work). On the design, needs date, author, and JIRA number. As a first step we only intend to limit the number of tables and regions a given namespace may contain. This can be done later, right? You'd have to do it in hbase. Are we saying the default ns is called 'default'? Or that it has no name? NamespaceController coprocessor running on region servers to correctly enforce quota restrictions Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own) How much of this design has to be in place for 0.96 to ship? <ROOT>/data/<NAMESPACE>/<TABLE> Why the need for 'data' in above? Looking quickly at the attached patch (will look at rb later): Below is interesting (smile) - <hadoop.version>0.23.3</hadoop.version> + <hadoop.version>0.23.7-SNAPSHOT</hadoop.version> Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off). NamespaceDescriptor implements java Serializable? NamespaceDesc.. has notion of groups. Is that right given they are add on for later? Will look at rb patch next. What is attached doesn't seem enough. Will be back.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

          Enis Soztutar do you have any bandwidth? Since you've shown interested earlier

          IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

          Looking at the comments there aren't any major issues apart from the delimiter. The namespace affected tests are passing.

          Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

          Either sounds fine to me.

          Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

          Working on my existing patch we can try out elliot's suggestion which would essentially require the new interfaces

          If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

          Yep it's possible. Though the patch will need some small changes in assignment manager and LoadBalancer interface, etc. Better to get those in soon if we're in a hurry. I can separate those out as a separate patch as well?

          Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

          It's gonna be hard, will give this a try.

          Show
          toffer Francis Liu added a comment - Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here) Enis Soztutar do you have any bandwidth? Since you've shown interested earlier IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this. Looking at the comments there aren't any major issues apart from the delimiter. The namespace affected tests are passing. Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going. Either sounds fine to me. Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out) Working on my existing patch we can try out elliot's suggestion which would essentially require the new interfaces If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above. Yep it's possible. Though the patch will need some small changes in assignment manager and LoadBalancer interface, etc. Better to get those in soon if we're in a hurry. I can separate those out as a separate patch as well? Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work). It's gonna be hard, will give this a try.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          This can be done later, right? You'd have to do it in hbase.

          Yep

          Are we saying the default ns is called 'default'? Or that it has no name?

          It has no name similar to the java default package.

          Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)

          I wish too. We can refactor after this to create one?

          How much of this design has to be in place for 0.96 to ship?

          We're watching out for backwards compatibility? I believe core namespace should be enough. We can put the rest in 96.1, quota stuff needs addition to the cli. Security has some changes as well to support namespaces but it's just an extension.

          Below is interesting (smile)

          It's our secret version

          Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).

          My line of thinking was that namespace is a component of tableName ie "tableName = <table namespace>.<table qualifier>". Just like column = CF:CQ.

          NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

          Where did you see that? Group information is added as a KV pair to the NamespaceDesc.

          NamespaceDescriptor implements java Serializable

          Oops will remove that

          Show
          toffer Francis Liu added a comment - This can be done later, right? You'd have to do it in hbase. Yep Are we saying the default ns is called 'default'? Or that it has no name? It has no name similar to the java default package. Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own) I wish too. We can refactor after this to create one? How much of this design has to be in place for 0.96 to ship? We're watching out for backwards compatibility? I believe core namespace should be enough. We can put the rest in 96.1, quota stuff needs addition to the cli. Security has some changes as well to support namespaces but it's just an extension. Below is interesting (smile) It's our secret version Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off). My line of thinking was that namespace is a component of tableName ie "tableName = <table namespace>.<table qualifier>". Just like column = CF:CQ. NamespaceDesc.. has notion of groups. Is that right given they are add on for later? Where did you see that? Group information is added as a KV pair to the NamespaceDesc. NamespaceDescriptor implements java Serializable Oops will remove that
          Hide
          Permalink
          stack stack added a comment -

          NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

          In patch attached here. May be gone in the rb version.

          Show
          stack stack added a comment - NamespaceDesc.. has notion of groups. Is that right given they are add on for later? In patch attached here. May be gone in the rb version.
          Hide
          Permalink
          stack stack added a comment -

          Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

          Show
          stack stack added a comment - Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).
          Hide
          Permalink
          toffer Francis Liu added a comment -

          RB patch is attached to subtask.

          Show
          toffer Francis Liu added a comment - RB patch is attached to subtask.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

          Our 0.94 patch is almost fully-baked so far so good...we already have it running in sandbox with users. Would a quick meetup with the people involved help iron things out?

          Show
          toffer Francis Liu added a comment - Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there). Our 0.94 patch is almost fully-baked so far so good...we already have it running in sandbox with users. Would a quick meetup with the people involved help iron things out?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Enis Soztutar do you have any bandwidth? Since you've shown interested earlier

          I am up for it.

          Would a quick meetup with the people involved help iron things out?

          +1. It is great that 0.94 version is in shape. The remaining questions are to handle migration, and the separator issue, and testing I guess.

          Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

          I am not a big fan of svn branches, but if it will help in reviews and testing, I can help with that.

          Show
          enis Enis Soztutar added a comment - Enis Soztutar do you have any bandwidth? Since you've shown interested earlier I am up for it. Would a quick meetup with the people involved help iron things out? +1. It is great that 0.94 version is in shape. The remaining questions are to handle migration, and the separator issue, and testing I guess. Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going. I am not a big fan of svn branches, but if it will help in reviews and testing, I can help with that.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          It seems we've come to a consensus on the upgrade approach.

          In essence it'll do the following:

          • Tables without '.' will be moved into the default namespace.
          • Tables with '.' will be move into new namespaces
          • namespaces will be delimited by the last '.' in the table name
            ie org.apache.hbase.MyTable, namespace = org.apache.hbase
          • In both cases the oldTableName is the same as the fullTableName
          • all existing apis and cli commands will be expecting full table names unless explicitly stated
          • a RenameTable tool will be provided to rename offline tables

          So I'll start implementing it.

          Discussion is here:
          http://grokbase.com/p/hbase/dev/13598jegtp/discuss-namespace-delimiter

          Show
          toffer Francis Liu added a comment - It seems we've come to a consensus on the upgrade approach. In essence it'll do the following: Tables without '.' will be moved into the default namespace. Tables with '.' will be move into new namespaces namespaces will be delimited by the last '.' in the table name ie org.apache.hbase.MyTable, namespace = org.apache.hbase In both cases the oldTableName is the same as the fullTableName all existing apis and cli commands will be expecting full table names unless explicitly stated a RenameTable tool will be provided to rename offline tables So I'll start implementing it. Discussion is here: http://grokbase.com/p/hbase/dev/13598jegtp/discuss-namespace-delimiter
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          a RenameTable tool will be provided to rename offline tables

          It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - a RenameTable tool will be provided to rename offline tables It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.
          Hide
          Permalink
          stack stack added a comment -

          It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

          No it shouldn't. There was no consensus on the list and this is an exploratory effort so no need of constraint.

          One thought I had was that if all tables need to be moved, mayhaps we just say snapshots are broke on upgrade or rather than a tool for table rename, instead, we'd have a tool that did surgery on snapshots to adjust them to match the new layout (advantage of this latter is that it can be done offline, not inline with migration – though, I have no idea this would work... ask the snapshotters Jon, Matteo and Jesse).

          Show
          stack stack added a comment - It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name. No it shouldn't. There was no consensus on the list and this is an exploratory effort so no need of constraint. One thought I had was that if all tables need to be moved, mayhaps we just say snapshots are broke on upgrade or rather than a tool for table rename, instead, we'd have a tool that did surgery on snapshots to adjust them to match the new layout (advantage of this latter is that it can be done offline, not inline with migration – though, I have no idea this would work... ask the snapshotters Jon, Matteo and Jesse).
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I've uploaded a first stab of the migration code on RB and HBASE-8408. Haven't addressed any of the comments yet so I could get this out first.

          I added a unit test which tests migration on a tarballed 0.94 rootdir. It does simple verification that the tables and snapshots have been migrated properly.

          Show
          toffer Francis Liu added a comment - I've uploaded a first stab of the migration code on RB and HBASE-8408 . Haven't addressed any of the comments yet so I could get this out first. I added a unit test which tests migration on a tarballed 0.94 rootdir. It does simple verification that the tables and snapshots have been migrated properly.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I believe the necessary code to verify this migration approach is in place. Please let me know if there are any concerns.

          Show
          toffer Francis Liu added a comment - I believe the necessary code to verify this migration approach is in place. Please let me know if there are any concerns.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Ran tests see below. "testDelayedRpcImmediateReturnValue" seems to be flakey. It passed when I reran the class separately.

          Small and medium (and large):

          Results :

          Tests in error:
          testClusterStatus(org.apache.hadoop.hbase.client.TestHCM): Unexpected exception, expected<org.apache.hadoop.hbase.exceptions.RegionServerStoppedException> but was<junit.framework.AssertionFailedError>
          testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details.
          testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Index: 1, Size: 1

          Tests run: 1494, Failures: 0, Errors: 3, Skipped: 13

          Large Tests:

          Results :

          Tests in error:
          testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details.

          Tests run: 512, Failures: 0, Errors: 1, Skipped: 11

          Show
          toffer Francis Liu added a comment - Ran tests see below. "testDelayedRpcImmediateReturnValue" seems to be flakey. It passed when I reran the class separately. Small and medium (and large): Results : Tests in error: testClusterStatus(org.apache.hadoop.hbase.client.TestHCM): Unexpected exception, expected<org.apache.hadoop.hbase.exceptions.RegionServerStoppedException> but was<junit.framework.AssertionFailedError> testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details. testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Index: 1, Size: 1 Tests run: 1494, Failures: 0, Errors: 3, Skipped: 13 Large Tests: Results : Tests in error: testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details. Tests run: 512, Failures: 0, Errors: 1, Skipped: 11
          Hide
          Permalink
          stack stack added a comment -

          What is state of this feature? What else needs to be done other than more review and test? Are we going to make a branch for it?

          Francis Liu How much testing of this feature has there been? Thanks boss.

          Show
          stack stack added a comment - What is state of this feature? What else needs to be done other than more review and test? Are we going to make a branch for it? Francis Liu How much testing of this feature has there been? Thanks boss.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          I am doing the review, but could definitely use more reviews. After the first patch (HBASE-8408) is +1'ed I will create a branch and commit there, since we will need the rest of the subtasks in shape before proposing the merge.

          Show
          enis Enis Soztutar added a comment - I am doing the review, but could definitely use more reviews. After the first patch ( HBASE-8408 ) is +1'ed I will create a branch and commit there, since we will need the rest of the subtasks in shape before proposing the merge.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Stack It's past testing and we're baking it in our sandbox environment where we have active users. No issues so far.

          Show
          toffer Francis Liu added a comment - Stack It's past testing and we're baking it in our sandbox environment where we have active users. No issues so far.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          As discussed I've created a github repo please see HBASE-8408. Please give it a try and let me know what you guys think.

          Show
          toffer Francis Liu added a comment - As discussed I've created a github repo please see HBASE-8408 . Please give it a try and let me know what you guys think.
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu Are we going to implement the Elliott suggestion adding overrides for namespaces? I was chatting w/ Nick Dimiduk last weds and he made a comment that helped: he said that if a choice between devs doing more work to save users 'surprise', then the answer is clear.

          Another thought is that we need to hurry up and get this in. I'll do a review now. Who else is reviewing?

          Thanks Francis.

          Show
          stack stack added a comment - Francis Liu Are we going to implement the Elliott suggestion adding overrides for namespaces? I was chatting w/ Nick Dimiduk last weds and he made a comment that helped: he said that if a choice between devs doing more work to save users 'surprise', then the answer is clear. Another thought is that we need to hurry up and get this in. I'll do a review now. Who else is reviewing? Thanks Francis.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          BTW regarding separator chars:
          "\/:*?<>|" NTFS does not allow these chars
          "\/~" Meaningful in ext/ntfs.
          "-" and "_" cannot be used because they are legal in table names.
          "#" Comment in many SQL dialects, not compatible with URL encoding.
          "$%&" Used in shell scripts

          This leaves: "!@^+=," if we disregard parenthesis chars. These also seem to work for HDFS. "," is our internal separator for region names and META naming. @ also seems good enough. Haven't tried these for znodes.

          It seems that we could not reach an agreement on which proposal to implement, where they are
          1) use dot as separator, autogenerate namespaces on migration
          2) use one of the candidate char's above as ns and table separator.
          3) In all places that require a table name, override with accepting namespace.

          However, it seems that we definitely have to have an internal separator char and a deterministic way of encoding <ns,table> pair into a string and decoding it back. This is because we need to do this for znodes in zktable, tablelockmanager, etc as well as META. So, I think it is a matter of whether we want to expose the separator char between proposal 2 and 3. I am in favor of doing 2, since it will allow better interop with external tools, and the code changes needed for 3 is significant.

          Show
          enis Enis Soztutar added a comment - BTW regarding separator chars: "\/:*?<>|" NTFS does not allow these chars "\/~" Meaningful in ext/ntfs. "-" and "_" cannot be used because they are legal in table names. "#" Comment in many SQL dialects, not compatible with URL encoding. "$%&" Used in shell scripts This leaves: "!@^+=," if we disregard parenthesis chars. These also seem to work for HDFS. "," is our internal separator for region names and META naming. @ also seems good enough. Haven't tried these for znodes. It seems that we could not reach an agreement on which proposal to implement, where they are 1) use dot as separator, autogenerate namespaces on migration 2) use one of the candidate char's above as ns and table separator. 3) In all places that require a table name, override with accepting namespace. However, it seems that we definitely have to have an internal separator char and a deterministic way of encoding <ns,table> pair into a string and decoding it back. This is because we need to do this for znodes in zktable, tablelockmanager, etc as well as META. So, I think it is a matter of whether we want to expose the separator char between proposal 2 and 3. I am in favor of doing 2, since it will allow better interop with external tools, and the code changes needed for 3 is significant.
          Hide
          Permalink
          stack stack added a comment -

          If we use '@', won't we have to swap the order so it is table 'at' namespace? (Or, the '@' would seem to imply that).

          Between 2 and 3, if we choose 3, can't we keep more hidden from the user – or allowing your 'external tools' argument, we should do 2 and 3?

          Show
          stack stack added a comment - If we use '@', won't we have to swap the order so it is table 'at' namespace? (Or, the '@' would seem to imply that). Between 2 and 3, if we choose 3, can't we keep more hidden from the user – or allowing your 'external tools' argument, we should do 2 and 3?
          Hide
          Permalink
          eclark Elliott Clark added a comment -

          Option 3 seems to involve less technical debt (or chance of tech debt).

          Show
          eclark Elliott Clark added a comment - Option 3 seems to involve less technical debt (or chance of tech debt).
          Hide
          Permalink
          stack stack added a comment -

          So, that leaves '^', the acute or hat character: namespace^table?

          Show
          stack stack added a comment - So, that leaves '^', the acute or hat character: namespace^table?
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          Option 4 is to use a dot, not create any automatic namespaces, have well-defined name resolution rules, and prohibit creating conflicts under those (also prohibit any further dots in table names for good measure). E.g. if user has org.my.cool.table, org.my.foo.table they go under default, and namespaces org, org.my, org.my.cool, and org.my.cool2 cannot be created at all until this table is renamed (or maybe they can be created by explicitly specifying that all tables should be renamed/moved under namespace keeping essentially the same name given the dots).
          That way it's explicit to the user what needs to be done (no shooting-in-the-foot); there are dots; and there's minimum surprise.
          The extra dev work is for mapping such table names to HDFS paths/etc.

          Show
          sershe Sergey Shelukhin added a comment - Option 4 is to use a dot, not create any automatic namespaces, have well-defined name resolution rules, and prohibit creating conflicts under those (also prohibit any further dots in table names for good measure). E.g. if user has org.my.cool.table, org.my.foo.table they go under default, and namespaces org, org.my, org.my.cool, and org.my.cool2 cannot be created at all until this table is renamed (or maybe they can be created by explicitly specifying that all tables should be renamed/moved under namespace keeping essentially the same name given the dots). That way it's explicit to the user what needs to be done (no shooting-in-the-foot); there are dots; and there's minimum surprise. The extra dev work is for mapping such table names to HDFS paths/etc.
          Hide
          Permalink
          stack stack added a comment -

          Option 4 would get us the '.' back (that would mean that we'd do options 3 and 4).

          Migrating, we will have a pre-update step where you run a script and it will check your install for v1 hfiles AND if you have tables with dots in them. If you have dots in your table name, you will be pointed at a page that describes the options available to you.

          Show
          stack stack added a comment - Option 4 would get us the '.' back (that would mean that we'd do options 3 and 4). Migrating, we will have a pre-update step where you run a script and it will check your install for v1 hfiles AND if you have tables with dots in them. If you have dots in your table name, you will be pointed at a page that describes the options available to you.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          I was thinking more along the lines of doing the non-surprising thing silently (if the user doesn't want to use namespaces he will never notice anything at all), and allowing the user to resolve conflicts as necessary while preventing them from unwittingly shooting himself in the foot; but migration-time options also work.

          Show
          sershe Sergey Shelukhin added a comment - I was thinking more along the lines of doing the non-surprising thing silently (if the user doesn't want to use namespaces he will never notice anything at all), and allowing the user to resolve conflicts as necessary while preventing them from unwittingly shooting himself in the foot; but migration-time options also work.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Stack, I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process.

          Show
          toffer Francis Liu added a comment - Stack , I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I thought of a way to implement Sergey Shelukhin idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo).

          Surprise here is:

          • you can't create tables with the delimiter no longer unless you create the appropriate namespace.
          • you can't create tables/namespace which conflict the exception tables/namespaces
          • the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive

          Here's a sample of how it works. I've updated the TestNamespaceUpgrade test to verify that it works:
          https://github.com/francisliu/hbase_namespace/tree/core_8408_exception_list

          Show
          toffer Francis Liu added a comment - I thought of a way to implement Sergey Shelukhin idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo). Surprise here is: you can't create tables with the delimiter no longer unless you create the appropriate namespace. you can't create tables/namespace which conflict the exception tables/namespaces the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive Here's a sample of how it works. I've updated the TestNamespaceUpgrade test to verify that it works: https://github.com/francisliu/hbase_namespace/tree/core_8408_exception_list
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Oops sorry I guess I'm talking about two scripts. One to check if some surprising migration needs to be done and provide links/options. And another that does the actual migration.

          Show
          toffer Francis Liu added a comment - Oops sorry I guess I'm talking about two scripts. One to check if some surprising migration needs to be done and provide links/options. And another that does the actual migration.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment - 
          exceptionNS.add(tableName.getNamespaceAsString());

          What is the current thinking on dots in namespaces and names? Presumably one table could prevent the creation of multiple namespaces if dots are allowed in namespace name, which I thought they are

          Show
          sershe Sergey Shelukhin added a comment - exceptionNS.add(tableName.getNamespaceAsString()); What is the current thinking on dots in namespaces and names? Presumably one table could prevent the creation of multiple namespaces if dots are allowed in namespace name, which I thought they are
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu There will be a migration evaluation script that will looks for presence of stuff like hfile v1s – they must be compacted away before you can upgrade – and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer).

          Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

          Did you get my review comments up on rb francis?

          On dots in namespace, no, if it simplifies.

          Show
          stack stack added a comment - Francis Liu There will be a migration evaluation script that will looks for presence of stuff like hfile v1s – they must be compacted away before you can upgrade – and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer). Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too. Did you get my review comments up on rb francis? On dots in namespace, no, if it simplifies.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

          The approach I proposed earlier would avoid having to do all the api stuff as part of the first namespace checkin as well as make use of '.' as a delimeter. The suprises are as I mentioned. We can incrementally add the apis.

          Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

          For some reason I'm not getting any jira or RB emails. Will take a look.

          Show
          toffer Francis Liu added a comment - Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too. The approach I proposed earlier would avoid having to do all the api stuff as part of the first namespace checkin as well as make use of '.' as a delimeter. The suprises are as I mentioned. We can incrementally add the apis. Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names? For some reason I'm not getting any jira or RB emails. Will take a look.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well.

          Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

          It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me.

          Show
          enis Enis Soztutar added a comment - One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well. Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names? It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me.
          Hide
          Permalink
          stack stack added a comment -

          If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

          I think the old API will be against default NS.

          The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

          Show
          stack stack added a comment - If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names? I think the old API will be against default NS. The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I think the old API will be against default NS.
          The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

          Is it possible to support FQTN as well? Or is going this route to avoid surprise completely? One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. As opposed to having just a table name then it's just a simple config change and restart.

          Show
          toffer Francis Liu added a comment - I think the old API will be against default NS. The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell). Is it possible to support FQTN as well? Or is going this route to avoid surprise completely? One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. As opposed to having just a table name then it's just a simple config change and restart.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

          I see, so what delimiter are we planning on using? '^' and '@' isn't appealing. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?

          Show
          toffer Francis Liu added a comment - The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell). I see, so what delimiter are we planning on using? '^' and '@' isn't appealing. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Also I'm wondering how this will affect the rest service. Will we need a v2 set of apis to support this?

          Show
          toffer Francis Liu added a comment - Also I'm wondering how this will affect the rest service. Will we need a v2 set of apis to support this?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          +thrift as well. That is why I am in favor of option 2 now.

          ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?

          I think we can use this. We are relying on it for partitioning region names, and in META. So it should be compatible with that framework.

          Show
          enis Enis Soztutar added a comment - +thrift as well. That is why I am in favor of option 2 now. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account? I think we can use this. We are relying on it for partitioning region names, and in META. So it should be compatible with that framework.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          We can broaden the range if we encode the delimiter when stored on fs and zk. We are mainly limited by region and table names.

          Show
          toffer Francis Liu added a comment - We can broaden the range if we encode the delimiter when stored on fs and zk. We are mainly limited by region and table names.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          ZK restrictions:

          http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#ch_zkDataModel

          If only they printed the chars instead of code points.

          Show
          toffer Francis Liu added a comment - ZK restrictions: http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#ch_zkDataModel If only they printed the chars instead of code points.
          Hide
          Permalink
          stack stack added a comment -

          One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS.

          You mean Y! They are using '.' for separator currently?

          As opposed to having just a table name then it's just a simple config change and restart.

          No override of APIs?

          We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','.

          I thought you had trick to make '.' work?

          Show
          stack stack added a comment - One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. You mean Y! They are using '.' for separator currently? As opposed to having just a table name then it's just a simple config change and restart. No override of APIs? We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','. I thought you had trick to make '.' work?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          You mean Y! They are using '.' for separator currently?

          The only dots we use are for namespaces and that's only in our sandbox so no worries there. My concern was that we'd have to ask users to upgrade their code in order to start using namespaces.

          So we'll have an internal delimiter '.' and an external one for external tools such as cli, rest, etc. But for the java apis we won't recognize an FQTN string in the public apis?

          I thought you had trick to make '.' work?

          Yes, I do. Enis Soztutar pointed out that we will have to live with that trick in later versions. Just making sure you guys are ok with that. Have a look at it on github, just look at TableName.java that should help you get the gist of it.

          Show
          toffer Francis Liu added a comment - You mean Y! They are using '.' for separator currently? The only dots we use are for namespaces and that's only in our sandbox so no worries there. My concern was that we'd have to ask users to upgrade their code in order to start using namespaces. So we'll have an internal delimiter '.' and an external one for external tools such as cli, rest , etc. But for the java apis we won't recognize an FQTN string in the public apis? I thought you had trick to make '.' work? Yes, I do. Enis Soztutar pointed out that we will have to live with that trick in later versions. Just making sure you guys are ok with that. Have a look at it on github, just look at TableName.java that should help you get the gist of it.
          Hide
          Permalink
          stack stack added a comment -

          The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?

          This is 0.96. Y! will have to do a 'migration' to go up on 0.96 anyways? Can you think of a 'step' that you could add for Y! cluster that would help here?

          Your users are doing 'new HTable(conf, "ns.sometable");' and away they go?

          But for the java apis we won't recognize an FQTN string in the public apis?

          Well, there would be helper classes for making sense of the FQTN that could be used by shell, rest, etc., but, yeah, the thought would be that in the API, there is no namepacing, not unless you ask for it explicitly.

          Let me look at the trick for how much we'd have to carry into the future.

          Show
          stack stack added a comment - The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out? This is 0.96. Y! will have to do a 'migration' to go up on 0.96 anyways? Can you think of a 'step' that you could add for Y! cluster that would help here? Your users are doing 'new HTable(conf, "ns.sometable");' and away they go? But for the java apis we won't recognize an FQTN string in the public apis? Well, there would be helper classes for making sense of the FQTN that could be used by shell, rest, etc., but, yeah, the thought would be that in the API, there is no namepacing, not unless you ask for it explicitly. Let me look at the trick for how much we'd have to carry into the future.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?

          We are not married to the current implementation yet. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

          Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

          What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

          *Nit side note: Shouldn't it be be called overload (not override), since we are planning on overloading a a method or am I thinking of one of the other approaches?

          Show
          toffer Francis Liu added a comment - The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out? We are not married to the current implementation yet. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem. Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override? What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string? *Nit side note: Shouldn't it be be called overload (not override), since we are planning on overloading a a method or am I thinking of one of the other approaches?
          Hide
          Permalink
          stack stack added a comment -

          Sorry, meant overload.

          But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

          The argument above is that the current implementation has been deployed some where so the current implementation is what we have to commit?

          Can we figure out something to bring along these legacy folks Francis? They will have to restart anyways going to the new stuff? Can we treat this problem apart from how ns goes into trunk/0.95?

          Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

          How would that work? How would we distingush a FQTN from a plain TN if the TN has the delimiter in it as part of its name. I can see helper methods that the shell or command-line tools could use but don't see these being run on every tablename we are passed.

          What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

          No conflation of namespace and tablename. Clean deliniation between the ns and tablename concepts. Users can take on the new namespace feature if they want to. No hackery.

          Chatting today, the '/' character came up again. Did we rule this out as a delimiter? Yeah '/' is meaningful in ext and ntfs but can't we leverage their interpretation; e.g. ns is a directory of tables? Ditto up in zk.

          Show
          stack stack added a comment - Sorry, meant overload. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem. The argument above is that the current implementation has been deployed some where so the current implementation is what we have to commit? Can we figure out something to bring along these legacy folks Francis? They will have to restart anyways going to the new stuff? Can we treat this problem apart from how ns goes into trunk/0.95? Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override? How would that work? How would we distingush a FQTN from a plain TN if the TN has the delimiter in it as part of its name. I can see helper methods that the shell or command-line tools could use but don't see these being run on every tablename we are passed. What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string? No conflation of namespace and tablename. Clean deliniation between the ns and tablename concepts. Users can take on the new namespace feature if they want to. No hackery. Chatting today, the '/' character came up again. Did we rule this out as a delimiter? Yeah '/' is meaningful in ext and ntfs but can't we leverage their interpretation; e.g. ns is a directory of tables? Ditto up in zk.
          Hide
          Permalink
          stack stack added a comment -

          '/' probably won't work. Or at least, it won't work up in zk because you want to have RS watch tables come and go (or is it NS come and go). If tables, you can't set a recursive watch so would seem to rule out '/'.

          Show
          stack stack added a comment - '/' probably won't work. Or at least, it won't work up in zk because you want to have RS watch tables come and go (or is it NS come and go). If tables, you can't set a recursive watch so would seem to rule out '/'.
          Hide
          Permalink
          sershe Sergey Shelukhin added a comment -

          To solve ambiguity create should have for FQ overload imo. Then, that means you always operate on existing tables if they have a dot, so you can just prevent conflicts at create time.

          Show
          sershe Sergey Shelukhin added a comment - To solve ambiguity create should have for FQ overload imo. Then, that means you always operate on existing tables if they have a dot, so you can just prevent conflicts at create time.
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu Status? I see no commits over in github?

          Show
          stack stack added a comment - Francis Liu Status? I see no commits over in github?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Stack Getting things to compile is taking more time than I thought. Fixing unit tests now. Should hoping to have something up today. Sorry for the delay.

          Show
          toffer Francis Liu added a comment - Stack Getting things to compile is taking more time than I thought. Fixing unit tests now. Should hoping to have something up today. Sorry for the delay.
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu Ping here so i can go take a looksee when you have something boss. Thanks.

          Show
          stack stack added a comment - Francis Liu Ping here so i can go take a looksee when you have something boss. Thanks.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Stack I've pushed changes into github, the changes mainly addresses one thing: make namespace as first class. I've replaced internal apis which took "byte[] tableName" as a constant to use FullyQualifiedTableName instead. For HTable and HBaseAdmin, I've overloaded such functions instead to keep backward compatibility. Also created a new branch in the repo which is a rebase off trunk yesterday. I'll address your RB comments today.

          Some issues:

          • The external interfaces: cli, thrift, rest and MR don't have the namespace as first class treatment yet
          • I initially went by the InterfaceAudience annotations but after talking to you guys it seems this is needs some updating as HConnection and HConnectionManager are both marked public tho they shouldn't be?
          • There are some apis (ie compact()) which have an overloaded parameter ie "byte[] tableOrRegionName" should we be splitting this into two apis and deprecate the old one?
          • Need to create a FullyQualifiedTableName PB equivalent and update PB rpc and messages as needed
          • I always needed to add commons-io as a maven dependency else things won't compile and these aren't related to my changes
          • Ran small,med,large tests and have these failures which aren't related to the patch: TestHCM.testClusterStatus, TestLogRolling.testCompactionRecordDoesntBlockRolling, TestShell
          Show
          toffer Francis Liu added a comment - Stack I've pushed changes into github, the changes mainly addresses one thing: make namespace as first class. I've replaced internal apis which took "byte[] tableName" as a constant to use FullyQualifiedTableName instead. For HTable and HBaseAdmin, I've overloaded such functions instead to keep backward compatibility. Also created a new branch in the repo which is a rebase off trunk yesterday. I'll address your RB comments today. Some issues: The external interfaces: cli, thrift, rest and MR don't have the namespace as first class treatment yet I initially went by the InterfaceAudience annotations but after talking to you guys it seems this is needs some updating as HConnection and HConnectionManager are both marked public tho they shouldn't be? There are some apis (ie compact()) which have an overloaded parameter ie "byte[] tableOrRegionName" should we be splitting this into two apis and deprecate the old one? Need to create a FullyQualifiedTableName PB equivalent and update PB rpc and messages as needed I always needed to add commons-io as a maven dependency else things won't compile and these aren't related to my changes Ran small,med,large tests and have these failures which aren't related to the patch: TestHCM.testClusterStatus, TestLogRolling.testCompactionRecordDoesntBlockRolling, TestShell
          Hide
          Permalink
          stack stack added a comment -

          How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?

          Yeah, HC/HCM should probably be evolving at least.

          On the tableOrRegionName, yeah, should probably a utility that is called to figure out which and then once that is known, the appropriate api is called.

          Not sure why the commons-io. Ditto on test failures. Yes, possible unrelated to your changes.

          Let me look at your work over in git.

          Show
          stack stack added a comment - How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable? Yeah, HC/HCM should probably be evolving at least. On the tableOrRegionName, yeah, should probably a utility that is called to figure out which and then once that is known, the appropriate api is called. Not sure why the commons-io. Ditto on test failures. Yes, possible unrelated to your changes. Let me look at your work over in git.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?

          It depends on how we'd wanna deal with it. Presently it's addressable as all the apis recognize FQTN names with '.' as the delimiter. If we go this route there won't be much work, just some cleanup to use FullyQualifiedTableName were needed.

          If we decide to have a different external and internal delimiter or overload the cli api there would be some amount of work.

          In either case could be a separate patch.

          Show
          toffer Francis Liu added a comment - How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable? It depends on how we'd wanna deal with it. Presently it's addressable as all the apis recognize FQTN names with '.' as the delimiter. If we go this route there won't be much work, just some cleanup to use FullyQualifiedTableName were needed. If we decide to have a different external and internal delimiter or overload the cli api there would be some amount of work. In either case could be a separate patch.
          Hide
          Permalink
          stack stack added a comment -

          I took a quick look over at https://github.com/francisliu/hbase_namespace/commit/18973af443e8c6424c978364e2d0db2a5abe1286#L5R705

          Patch is massive but seems to be in the right direction. I'm thinking weeks – three to four weeks – for this to stabilize and get committed?

          Show
          stack stack added a comment - I took a quick look over at https://github.com/francisliu/hbase_namespace/commit/18973af443e8c6424c978364e2d0db2a5abe1286#L5R705 Patch is massive but seems to be in the right direction. I'm thinking weeks – three to four weeks – for this to stabilize and get committed?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          It's massive but not really complex. What does stabilizing involve? Reviews and all relevant unit tests passing?

          Show
          toffer Francis Liu added a comment - It's massive but not really complex. What does stabilizing involve? Reviews and all relevant unit tests passing?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          BTW would it help if I put the patch on RB as well? 309 files might be too much.

          Show
          toffer Francis Liu added a comment - BTW would it help if I put the patch on RB as well? 309 files might be too much.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I had a chat with Stack yesterday to iron out some design details. Here's a quick summary of things:

          1. We will be going with ':' as the delimiter. We'll make it so that the tableNames stored on the filesystem don't get stored fully-qualified. So that leaves HFileLinks which needs table names to be written with the namespace.

          2. a TableName POJO will be the class that gets passed around where a tableName reference is needed. HTable and HBaseAdmin will have the old apis overloaded with new apis that accept the TableName pojo. In addition, the old apis will recognize fully qualified table names. This keeps symmetry between both apis. Having colon as a delimiter will guarantee there is no confusion and clear intent.

          3. Since we will be using a TableName pojo everywhere internally. That means all the coprocessor hooks that have tableNames will change as well. This will break backward compatibility.

          4. External interfaces REST, Thrift and CLI will not require significant changes as tables can be referenced using fully-qualified. We will have to add namespace CRUD apis to REST and Thrift at a latter point.

          5. In a separate patch remove '.' prefix in non-table dirs in root dir

          Show
          toffer Francis Liu added a comment - I had a chat with Stack yesterday to iron out some design details. Here's a quick summary of things: 1. We will be going with ':' as the delimiter. We'll make it so that the tableNames stored on the filesystem don't get stored fully-qualified. So that leaves HFileLinks which needs table names to be written with the namespace. 2. a TableName POJO will be the class that gets passed around where a tableName reference is needed. HTable and HBaseAdmin will have the old apis overloaded with new apis that accept the TableName pojo. In addition, the old apis will recognize fully qualified table names. This keeps symmetry between both apis. Having colon as a delimiter will guarantee there is no confusion and clear intent. 3. Since we will be using a TableName pojo everywhere internally. That means all the coprocessor hooks that have tableNames will change as well. This will break backward compatibility. 4. External interfaces REST, Thrift and CLI will not require significant changes as tables can be referenced using fully-qualified. We will have to add namespace CRUD apis to REST and Thrift at a latter point. 5. In a separate patch remove '.' prefix in non-table dirs in root dir
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Forgot to mention, ':' is not a valid character on NTFS. So we'd have to have a different separate for full table names in HFileLinks. Possibly a switch to support this. Thoughts Enis Soztutar?

          Show
          toffer Francis Liu added a comment - Forgot to mention, ':' is not a valid character on NTFS. So we'd have to have a different separate for full table names in HFileLinks. Possibly a switch to support this. Thoughts Enis Soztutar ?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          We can use a different separator for file system then we expose to the users. Haven't checked whether zk also allows ":". Using comma for this makes sense if you ask me.

          Show
          enis Enis Soztutar added a comment - We can use a different separator for file system then we expose to the users. Haven't checked whether zk also allows ":". Using comma for this makes sense if you ask me.
          Hide
          Permalink
          stack stack added a comment -

          Comma is special in .META. table. Would have to adjust comparators. It would take us further away from one day being able to do memcmp on .META. rows keys.

          Show
          stack stack added a comment - Comma is special in .META. table. Would have to adjust comparators. It would take us further away from one day being able to do memcmp on .META. rows keys.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          @enis it looks like ':' is not in this list of not-allowed characters in ZK. I naively tried it and it works. Just curious I'm not familiar with the HBase on windows effort. They won't be running HBase on HDFS? But on a windows DFS?

          @stack I believe enis is suggesting on using a different delimiter when storing FQTN as part of filenames on hdfs.

          Show
          toffer Francis Liu added a comment - @enis it looks like ':' is not in this list of not-allowed characters in ZK. I naively tried it and it works. Just curious I'm not familiar with the HBase on windows effort. They won't be running HBase on HDFS? But on a windows DFS? @stack I believe enis is suggesting on using a different delimiter when storing FQTN as part of filenames on hdfs.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Yes, we can use different separator chars for fqtn depending on context. META and other places will use colon, while filesystem might use some other char (possibly comma)

          Show
          enis Enis Soztutar added a comment - Yes, we can use different separator chars for fqtn depending on context. META and other places will use colon, while filesystem might use some other char (possibly comma)
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Just re-read your comments for not storing FQ name on file system. Then we can just do this for file link which already contains carefully-selected separator chars for different fields.

          Show
          enis Enis Soztutar added a comment - Just re-read your comments for not storing FQ name on file system. Then we can just do this for file link which already contains carefully-selected separator chars for different fields.
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu Any status boss?

          Show
          stack stack added a comment - Francis Liu Any status boss?
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          Played with this for some time. Overall, shell commands, zookeeper data, ns table, hdfs data seems to be working. Did a skim through the commits at github. Waiting for the separator change.

          A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)

          Show
          enis Enis Soztutar added a comment - Played with this for some time. Overall, shell commands, zookeeper data, ns table, hdfs data seems to be working. Did a skim through the commits at github. Waiting for the separator change. A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I've pushed some changes into github addressing the following:

          • Change delimiter to ':'
          • Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name
          • changed filesystem table path to: <root>/data/<ns>/<table qualifier>
          • updated namespace and table name valid chars. namespace now only allows [A-Za-z0-9_]
          • rename FullyQualifiedTableName to TableName since all table names are fully qualified not unless explicitly stated

          Some issues:

          • found out that ':' is not a valid character in HDFS (scratches head). so we'll really need an fs delimiter if we stick with this delimiter.
          • snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it
          • small and medium tests pass, didn't have time to run large yet

          What do you guys think? I'll continue working on the other comments.

          Check out the changes here:

          https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1

          Show
          toffer Francis Liu added a comment - I've pushed some changes into github addressing the following: Change delimiter to ':' Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name changed filesystem table path to: <root>/data/<ns>/<table qualifier> updated namespace and table name valid chars. namespace now only allows [A-Za-z0-9_] rename FullyQualifiedTableName to TableName since all table names are fully qualified not unless explicitly stated Some issues: found out that ':' is not a valid character in HDFS (scratches head). so we'll really need an fs delimiter if we stick with this delimiter. snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it small and medium tests pass, didn't have time to run large yet What do you guys think? I'll continue working on the other comments. Check out the changes here: https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1
          Hide
          Permalink
          toffer Francis Liu added a comment -

          A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)

          Is meta public?

          Show
          toffer Francis Liu added a comment - A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now) Is meta public?
          Hide
          Permalink
          jesse_yates Jesse Yates added a comment -

          snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it

          It used to follow table name semantics, but looks like it just tries on the FS and fails if its not a valid character.

          Show
          jesse_yates Jesse Yates added a comment - snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it It used to follow table name semantics, but looks like it just tries on the FS and fails if its not a valid character.
          Hide
          Permalink
          enis Enis Soztutar added a comment -

          I think above scheme is acceptable and the best we can do right now. ns:table also fits into cf:column naming as well.

          rename FullyQualifiedTableName to TableName

          makes sense.

          Show
          enis Enis Soztutar added a comment - I think above scheme is acceptable and the best we can do right now. ns:table also fits into cf:column naming as well. rename FullyQualifiedTableName to TableName makes sense.
          Hide
          Permalink
          stack stack added a comment -

          Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name

          Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?

          Presently snapshot creation will complain if it has a ':' in it

          When you create a snapshot, can you use a TableName?

          Show
          stack stack added a comment - Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir? Presently snapshot creation will complain if it has a ':' in it When you create a snapshot, can you use a TableName?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?

          There's currently there are three places right now.

          1. HFileLink
          a. HFile references require include the table name as part of the filename
          b. back references include table name as well
          2. Snapshot names
          a. In current behavior as Jesse Yates pointed out. Valid table names are valid snapshot names. Snapshots are stored on hdfs as a directory identified by their names. Though I'm inclined not to support ':' as a valid character for snapshot names so we can keep ':' as a special character?

          Show
          toffer Francis Liu added a comment - Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir? There's currently there are three places right now. 1. HFileLink a. HFile references require include the table name as part of the filename b. back references include table name as well 2. Snapshot names a. In current behavior as Jesse Yates pointed out. Valid table names are valid snapshot names. Snapshots are stored on hdfs as a directory identified by their names. Though I'm inclined not to support ':' as a valid character for snapshot names so we can keep ':' as a special character?
          Hide
          Permalink
          stack stack added a comment -

          On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? Could use current link separator for added ns? hfilelink already has its own notion of separator?

          On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

          3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

          So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

          Will review github.

          What is left to finish would you say?

          Show
          stack stack added a comment - On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? Could use current link separator for added ns? hfilelink already has its own notion of separator? On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys) 3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names) So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir). Will review github. What is left to finish would you say?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          On 1., you mean existing HFileLinks? They would be to tables in default namespace?
          Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name?

          For migrated tables yes. But for new tables which use NS they could be to new NS tables so we need to add NS as part of the HFileLink. I've updated the HFileLink file format to look like <NS>,<TableQualifier><...><...> it will also recognize the older file format where '<NS>,' is missing (See HFileLink.java for more details). Are you suggesting I change it to <TableQualifier><...><...>-<NS>?

          Could use current link separator for added ns? hfilelink already has its own notion of separator?

          Yep, currently I'm using ','. I could use '' instead. So it becomes <NS><TableQualifier><...><...>?

          On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

          Sorry I meant the HFileLink Back references, not region split references. I think it's used for quick reference counting to see if an archived HFile can be cleaned up or not.

          3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

          Cool will add the check.

          So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

          Yep, as far as I can tell. I've ran small, medium and large tests, it only had unrelated failures.

          What is left to finish would you say?

          • Add a TableName PB
          • Your comments on RB

          That's about it.

          Show
          toffer Francis Liu added a comment - On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? For migrated tables yes. But for new tables which use NS they could be to new NS tables so we need to add NS as part of the HFileLink. I've updated the HFileLink file format to look like <NS>,<TableQualifier> <...> <...> it will also recognize the older file format where '<NS>,' is missing (See HFileLink.java for more details). Are you suggesting I change it to <TableQualifier> <...> <...>-<NS>? Could use current link separator for added ns? hfilelink already has its own notion of separator? Yep, currently I'm using ','. I could use ' ' instead. So it becomes <NS> <TableQualifier> <...> <...>? On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys) Sorry I meant the HFileLink Back references, not region split references. I think it's used for quick reference counting to see if an archived HFile can be cleaned up or not. 3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names) Cool will add the check. So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir). Yep, as far as I can tell. I've ran small, medium and large tests, it only had unrelated failures. What is left to finish would you say? Add a TableName PB Your comments on RB That's about it.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Could use current link separator for added ns? hfilelink already has its own notion of separator?

          Forgot to mention for HFileLink it uses hyphens and for backreferences it uses dots.

          Show
          toffer Francis Liu added a comment - Could use current link separator for added ns? hfilelink already has its own notion of separator? Forgot to mention for HFileLink it uses hyphens and for backreferences it uses dots.
          Hide
          Permalink
          stack stack added a comment -

          Are you suggesting I change it to <TableQualifier><...><...>-<NS>?

          I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)

          Your comments on RB

          You updated rb?

          Show
          stack stack added a comment - Are you suggesting I change it to <TableQualifier><...><...>-<NS>? I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?) Your comments on RB You updated rb?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)

          Yep got that with your second question. Will do.

          You updated rb?

          Not yet. Will create a new RB once I've put in all the missing changes as you suggested.

          Show
          toffer Francis Liu added a comment - I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?) Yep got that with your second question. Will do. You updated rb? Not yet. Will create a new RB once I've put in all the missing changes as you suggested.
          Hide
          Permalink
          stack stack added a comment -

          Francis Liu Thanks (in case you've not noticed – smile – time is running out for getting this in)

          Show
          stack stack added a comment - Francis Liu Thanks (in case you've not noticed – smile – time is running out for getting this in)
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Yes, working on it. I'll try to put the final patch for review EOD today.

          Show
          toffer Francis Liu added a comment - Yes, working on it. I'll try to put the final patch for review EOD today.
          Hide
          Permalink
          toffer Francis Liu added a comment -

          RB of near finished patch. It's missing some comments from stack in previous RB.

          https://reviews.apache.org/r/13178/

          Show
          toffer Francis Liu added a comment - RB of near finished patch. It's missing some comments from stack in previous RB. https://reviews.apache.org/r/13178/
          Hide
          Permalink
          toffer Francis Liu added a comment -

          github repo of patch:

          https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_2

          Show
          toffer Francis Liu added a comment - github repo of patch: https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_2
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Mind attaching the tar ball (src/test/data/TestNamespaceUpgrade.tgz) used by TestNamespaceUpgrade to the JIRA (with brief description on how it was generated) ?

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Mind attaching the tar ball (src/test/data/TestNamespaceUpgrade.tgz) used by TestNamespaceUpgrade to the JIRA (with brief description on how it was generated) ?
          Hide
          Permalink
          toffer Francis Liu added a comment -

          Yep it's attached to HBASE-8408. Let me know if you're having trouble with it. The brief description is written in TestNamespaceUpgrade. See the source file for more info:

          /**

          • Test upgrade from no namespace in 0.94 to namespace directory structure.
          • Mainly tests that tables are migrated and consistent. Also verifies
          • that snapshots have been migrated correctly.
            *
          • Uses a tarball which is an image of an 0.94 hbase.rootdir.
            *
          • Contains tables with currentKeys as the stored keys:
          • foo, ns1.foo, ns2.foo
            *
          • Contains snapshots with snapshot {num}

            Keys as the contents:

          • snapshot1Keys, snapshot2Keys
            *
            */
          Show
          toffer Francis Liu added a comment - Yep it's attached to HBASE-8408 . Let me know if you're having trouble with it. The brief description is written in TestNamespaceUpgrade. See the source file for more info: /** Test upgrade from no namespace in 0.94 to namespace directory structure. Mainly tests that tables are migrated and consistent. Also verifies that snapshots have been migrated correctly. * Uses a tarball which is an image of an 0.94 hbase.rootdir. * Contains tables with currentKeys as the stored keys: foo, ns1.foo, ns2.foo * Contains snapshots with snapshot {num} Keys as the contents: snapshot1Keys, snapshot2Keys * */
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment -

          Got it. The test passed.

          I think the untar() method can be moved to util class so that code is shared with TestMetaMigrationConvertingToPB.

          +      String srcTable = newNS+TableName.NAMESPACE_DELIM+table.replaceAll("[.]","_")+"_clone3";

          The replaceAll() call only appears in this test. Can you tell me the reason ?

          --- /dev/null
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java

          I think the above class was removed by HBASE-8764. Please refresh your workspace and update your patch.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Got it. The test passed. I think the untar() method can be moved to util class so that code is shared with TestMetaMigrationConvertingToPB. + String srcTable = newNS+TableName.NAMESPACE_DELIM+table.replaceAll( "[.]" , "_" )+ "_clone3" ; The replaceAll() call only appears in this test. Can you tell me the reason ? --- /dev/ null +++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java I think the above class was removed by HBASE-8764 . Please refresh your workspace and update your patch.
          Hide
          Permalink
          yuzhihong@gmail.com Ted Yu added a comment - 
          +  public static final byte [] DEFAULT_NAMESPACE_NAME = Bytes.toBytes("default");

          What if user already had table named 'default.x' ?

          Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ?

          TestRestoreFlushSnapshotFromClient seems to hang:

          "main" prio=10 tid=0x00007ffb7c007800 nid=0x28a4 waiting on condition [0x00007ffb82112000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
  at java.lang.Thread.sleep(Native Method)
  at org.apache.hadoop.hbase.client.HBaseAdmin.waitUntilTableIsEnabled(HBaseAdmin.java:734)
  at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2725)
  at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2674)
  at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshotOfCloned(TestRestoreFlushSnapshotFromClient.java:210)

          In test output, I saw:

          2013-08-01 03:01:47,059 ERROR [RS_OPEN_REGION-kiyo:35035-0] handler.OpenRegionHandler(475): Failed open of region=clonedtb-1375326090526,c,1375326056050.2fffb17b7a45e6f20fd296ffe0b1a3e7., starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:692)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:608)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:579)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4111)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4082)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4031)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3982)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:459)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:137)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:448)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:241)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3093)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:665)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:662)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        ... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
          Show
          yuzhihong@gmail.com Ted Yu added a comment - + public static final byte [] DEFAULT_NAMESPACE_NAME = Bytes.toBytes( " default " ); What if user already had table named 'default.x' ? Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ? TestRestoreFlushSnapshotFromClient seems to hang: "main" prio=10 tid=0x00007ffb7c007800 nid=0x28a4 waiting on condition [0x00007ffb82112000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.hbase.client.HBaseAdmin.waitUntilTableIsEnabled(HBaseAdmin.java:734) at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2725) at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2674) at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshotOfCloned(TestRestoreFlushSnapshotFromClient.java:210) In test output, I saw: 2013-08-01 03:01:47,059 ERROR [RS_OPEN_REGION-kiyo:35035-0] handler.OpenRegionHandler(475): Failed open of region=clonedtb-1375326090526,c,1375326056050.2fffb17b7a45e6f20fd296ffe0b1a3e7., starting to roll back the global memstore size. java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs: //localhost:39415/user/hortonzy/hbase/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d] at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:692) at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:608) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:579) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4111) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4082) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4031) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3982) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:459) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:137) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs: //localhost:39415/user/hortonzy/hbase/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d] at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:448) at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:241) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3093) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:665) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:662) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs: //localhost:39415/user/hortonzy/hbase/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/ default /testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
          Hide
          Permalink
          toffer Francis Liu added a comment -

          What if user already had table named 'default.x' ?

          Then the path on hdfs will be <ROOT_DIR>/default/default.x

          This is no longer an issue since we changed the delimiter to ':'. So the scenario where a table which already uses the delimiter no longer exists.

          Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ?

          Yep that image contains tables like that: ns1.foo and ns2.foo

          Show
          toffer Francis Liu added a comment - What if user already had table named 'default.x' ? Then the path on hdfs will be <ROOT_DIR>/default/default.x This is no longer an issue since we changed the delimiter to ':'. So the scenario where a table which already uses the delimiter no longer exists. Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ? Yep that image contains tables like that: ns1.foo and ns2.foo
          Hide
          Permalink
          toffer Francis Liu added a comment -

          The replaceAll() call only appears in this test. Can you tell me the reason ?

          It was needed to test snapshot across namespace when we had '.' as the delimiter. I've removed replaceAll() in the new patch.

          I think the above class was removed by HBASE-8764. Please refresh your workspace and update your patch.

          Re-removed it and a number of other files. I think it stayed during merge because I modified those files.

          TestRestoreFlushSnapshotFromClient seems to hang:

          Fixed this seemed to be a typo during cleanup.

          Show
          toffer Francis Liu added a comment - The replaceAll() call only appears in this test. Can you tell me the reason ? It was needed to test snapshot across namespace when we had '.' as the delimiter. I've removed replaceAll() in the new patch. I think the above class was removed by HBASE-8764 . Please refresh your workspace and update your patch. Re-removed it and a number of other files. I think it stayed during merge because I modified those files. TestRestoreFlushSnapshotFromClient seems to hang: Fixed this seemed to be a typo during cleanup.
          Hide
          Permalink
          enis Enis Soztutar added a comment - - edited

          Created HBASE-9134 for running the NS upgrade script from migration script.

          Show
          enis Enis Soztutar added a comment - - edited Created HBASE-9134 for running the NS upgrade script from migration script.
          Hide
          Permalink
          stack stack added a comment -

          Enis Soztutar Where you at on your review? You want an update of the patch by Francis Liu? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you Enis Soztutar, I could back out HBASE-8778 because it would be easier applying that on top of ns.

          Show
          stack stack added a comment - Enis Soztutar Where you at on your review? You want an update of the patch by Francis Liu ? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you Enis Soztutar , I could back out HBASE-8778 because it would be easier applying that on top of ns.
          Hide
          Permalink
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #4355 (See https://builds.apache.org/job/HBase-TRUNK/4355/)
          HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589)

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #4355 (See https://builds.apache.org/job/HBase-TRUNK/4355/ ) HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java
          Hide
          Permalink
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/)
          HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589)

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/ ) HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java
          Hide
          Permalink
          prasasf Prasanna added a comment -

          Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?

          How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?

          I would like namespace to be totally logical separation all sharing the same region file. Is this case?
          Thank you for any responses.

          Show
          prasasf Prasanna added a comment - Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files? How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale? I would like namespace to be totally logical separation all sharing the same region file. Is this case? Thank you for any responses.
          Hide
          Permalink
          apurtell Andrew Purtell added a comment -

          Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?

          Yes

          How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?

          As you would anticipate, eventually not, depending on how many tables each user creates and the capabilities of the namenode. I believe the intent of namespaces is to support multitenancy where a tenant shares a namespace and is a department or similar grouping within a larger organization. The namespace is a supergroup of tables sharing common ACLs, quotas, and the like.

          I would like namespace to be totally logical separation all sharing the same region file. Is this case?

          No it is not and this would be a substantial change.

          Show
          apurtell Andrew Purtell added a comment - Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files? Yes How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale? As you would anticipate, eventually not, depending on how many tables each user creates and the capabilities of the namenode. I believe the intent of namespaces is to support multitenancy where a tenant shares a namespace and is a department or similar grouping within a larger organization. The namespace is a supergroup of tables sharing common ACLs, quotas, and the like. I would like namespace to be totally logical separation all sharing the same region file. Is this case? No it is not and this would be a substantial change.
          Hide
          Permalink
          shihaoliang Peter Shi added a comment -

          Can i have namespace in the specified HDFS folder?
          Because in our multi-tenant scenarios, all the use resources are under the folder, files and hive tables and HBase datas.
          We want to control all the data size in HDFS quota on the folder.

          My propose is to have configurations for Namespace to modified the default path.

          Show
          shihaoliang Peter Shi added a comment - Can i have namespace in the specified HDFS folder? Because in our multi-tenant scenarios, all the use resources are under the folder, files and hive tables and HBase datas. We want to control all the data size in HDFS quota on the folder. My propose is to have configurations for Namespace to modified the default path.
          Hide
          Permalink
          shashi.vish Shashi added a comment -

          Can I export a namespace from one hbase cluster to another hbase cluster? Similar way that we can use Export/Import command with hbase table.

          Show
          shashi.vish Shashi added a comment - Can I export a namespace from one hbase cluster to another hbase cluster? Similar way that we can use Export/Import command with hbase table.
          Hide
          Permalink
          kartik kartik added a comment -

          I would like to know if this JIRA enhancement also support below four items using namespace ?

          Cell Query (Single Value)
          Cell or Row Query (Multiple Values)
          Scanner Get Next
          Scanner Creation

          Regards,
          Kartik Bhatnagar

          Show
          kartik kartik added a comment - I would like to know if this JIRA enhancement also support below four items using namespace ? Cell Query (Single Value) Cell or Row Query (Multiple Values) Scanner Get Next Scanner Creation Regards, Kartik Bhatnagar
          Hide
          Permalink
          jmspaggi Jean-Marc Spaggiari added a comment -

          Hi Kartik,

          You are better ask your questions on the mailing list.

          JM

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Hi Kartik, You are better ask your questions on the mailing list. JM

            People

            • Assignee:
              toffer Francis Liu
              Reporter:
              toffer Francis Liu
            • Votes:
              3 Vote for this issue
              Watchers:
              39 Start watching this issue

              Dates

              • Created:
                Updated:

                Development