Hi Kartik,

You are better ask your questions on the mailing list.

JM

I would like to know if this JIRA enhancement also support below four items using namespace ?

Cell Query (Single Value)
Cell or Row Query (Multiple Values)
Scanner Get Next
Scanner Creation

Regards,
Kartik Bhatnagar

Can I export a namespace from one hbase cluster to another hbase cluster? Similar way that we can use Export/Import command with hbase table.

Can i have namespace in the specified HDFS folder?
Because in our multi-tenant scenarios, all the use resources are under the folder, files and hive tables and HBase datas.
We want to control all the data size in HDFS quota on the folder.

My propose is to have configurations for Namespace to modified the default path.

Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?

Yes

How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?

As you would anticipate, eventually not, depending on how many tables each user creates and the capabilities of the namenode. I believe the intent of namespaces is to support multitenancy where a tenant shares a namespace and is a department or similar grouping within a larger organization. The namespace is a supergroup of tables sharing common ACLs, quotas, and the like.

I would like namespace to be totally logical separation all sharing the same region file. Is this case?

No it is not and this would be a substantial change.

Do different namespaces for the same table name create different tables directories in hdfs and are maintained in different region files?

How does this affect namenode memory if I want to have one namespace per user and I have a million users registered to my system. Will this scale?

I would like namespace to be totally logical separation all sharing the same region file. Is this case?
Thank you for any responses.

FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #657 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/657/)
HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589)

• /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java

FAILURE: Integrated in HBase-TRUNK #4355 (See https://builds.apache.org/job/HBase-TRUNK/4355/)
HBASE-8015 implement namespaces; ADDENDUM TWO – REMOVE FILE REMOVED BY ORIGINAL PATCH (stack: rev 1511589)

• /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TableName.java

Enis Soztutar Where you at on your review? You want an update of the patch by Francis Liu? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you Enis Soztutar, I could back out HBASE-8778 because it would be easier applying that on top of ns.

Created HBASE-9134 for running the NS upgrade script from migration script.

The replaceAll() call only appears in this test. Can you tell me the reason ?

It was needed to test snapshot across namespace when we had '.' as the delimiter. I've removed replaceAll() in the new patch.

I think the above class was removed by HBASE-8764. Please refresh your workspace and update your patch.

Re-removed it and a number of other files. I think it stayed during merge because I modified those files.

TestRestoreFlushSnapshotFromClient seems to hang:

Fixed this seemed to be a typo during cleanup.

What if user already had table named 'default.x' ?

Then the path on hdfs will be <ROOT_DIR>/default/default.x

This is no longer an issue since we changed the delimiter to ':'. So the scenario where a table which already uses the delimiter no longer exists.

Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ?

Yep that image contains tables like that: ns1.foo and ns2.foo

+  public static final byte [] DEFAULT_NAMESPACE_NAME = Bytes.toBytes("default");

What if user already had table named 'default.x' ?

Can you add some tables named 'a.b' in the 0.94 cluster image in the migration test ?

TestRestoreFlushSnapshotFromClient seems to hang:

"main" prio=10 tid=0x00007ffb7c007800 nid=0x28a4 waiting on condition [0x00007ffb82112000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
  at java.lang.Thread.sleep(Native Method)
  at org.apache.hadoop.hbase.client.HBaseAdmin.waitUntilTableIsEnabled(HBaseAdmin.java:734)
  at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2725)
  at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2674)
  at org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient.testRestoreSnapshotOfCloned(TestRestoreFlushSnapshotFromClient.java:210)

In test output, I saw:

2013-08-01 03:01:47,059 ERROR [RS_OPEN_REGION-kiyo:35035-0] handler.OpenRegionHandler(475): Failed open of region=clonedtb-1375326090526,c,1375326056050.2fffb17b7a45e6f20fd296ffe0b1a3e7., starting to roll back the global memstore size.
java.io.IOException: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:692)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:608)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:579)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4111)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4082)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4031)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3982)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:459)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:137)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]
        at org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:448)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:241)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3093)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:665)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:662)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        ... 3 more
Caused by: java.io.FileNotFoundException: Unable to open link: org.apache.hadoop.hbase.io.HFileLink locations=[hdfs://localhost:39415/user/hortonzy/hbase/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.tmp/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d, hdfs://localhost:39415/user/hortonzy/hbase/.archive/data/default/testtb-1375326056044/ee59283afb6da8861c97fda630c2571d/cf/ee59283afb6da8861c97fda630c2571d]

Got it. The test passed.

I think the untar() method can be moved to util class so that code is shared with TestMetaMigrationConvertingToPB.

+      String srcTable = newNS+TableName.NAMESPACE_DELIM+table.replaceAll("[.]","_")+"_clone3";

The replaceAll() call only appears in this test. Can you tell me the reason ?

--- /dev/null
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java

I think the above class was removed by HBASE-8764. Please refresh your workspace and update your patch.

Yep it's attached to HBASE-8408. Let me know if you're having trouble with it. The brief description is written in TestNamespaceUpgrade. See the source file for more info:

/**

• Test upgrade from no namespace in 0.94 to namespace directory structure.
• Mainly tests that tables are migrated and consistent. Also verifies
• that snapshots have been migrated correctly.
  *
• Uses a tarball which is an image of an 0.94 hbase.rootdir.
  *
• Contains tables with currentKeys as the stored keys:
• foo, ns1.foo, ns2.foo
  *
• Contains snapshots with snapshot {num}

  Keys as the contents:

• snapshot1Keys, snapshot2Keys
  *
  */

Mind attaching the tar ball (src/test/data/TestNamespaceUpgrade.tgz) used by TestNamespaceUpgrade to the JIRA (with brief description on how it was generated) ?

github repo of patch:

https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_2

RB of near finished patch. It's missing some comments from stack in previous RB.

https://reviews.apache.org/r/13178/

Yes, working on it. I'll try to put the final patch for review EOD today.

Francis Liu Thanks (in case you've not noticed – smile – time is running out for getting this in)

I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)

Yep got that with your second question. Will do.

You updated rb?

Not yet. Will create a new RB once I've put in all the missing changes as you suggested.

Are you suggesting I change it to <TableQualifier><...><...>-<NS>?

I was suggesting that NS be delimited the same way table and region are currently delimited rather than intro new delimiter ','; i.e. hypens and dots (as applicable?)

Your comments on RB

You updated rb?

Could use current link separator for added ns? hfilelink already has its own notion of separator?

Forgot to mention for HFileLink it uses hyphens and for backreferences it uses dots.

On 1., you mean existing HFileLinks? They would be to tables in default namespace?
Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name?

For migrated tables yes. But for new tables which use NS they could be to new NS tables so we need to add NS as part of the HFileLink. I've updated the HFileLink file format to look like <NS>,<TableQualifier><...><...> it will also recognize the older file format where '<NS>,' is missing (See HFileLink.java for more details). Are you suggesting I change it to <TableQualifier><...><...>-<NS>?

Could use current link separator for added ns? hfilelink already has its own notion of separator?

Yep, currently I'm using ','. I could use '' instead. So it becomes <NS><TableQualifier><...><...>?

On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

Sorry I meant the HFileLink Back references, not region split references. I think it's used for quick reference counting to see if an archived HFile can be cleaned up or not.

3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

Cool will add the check.

So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

Yep, as far as I can tell. I've ran small, medium and large tests, it only had unrelated failures.

What is left to finish would you say?

• Add a TableName PB
• Your comments on RB

That's about it.

On 1., you mean existing HFileLinks? They would be to tables in default namespace? Can HFileLink be updated to look there and then add namespace in front of tablename, regioname and filename when creating the name? Could use current link separator for added ns? hfilelink already has its own notion of separator?

On 2., references are always 'local' to a table – won't be across namespaces? Where in HFile reference do we reference table name? (I see regionnames and splitkeys)

3. On snapshot names, yeah, go ahead and make ':' special character (not allowed in snapshot names)

So would the above be the only places we'd want to persist a ':' in fs? (In fs we'd usually ahve table in a ns subdir).

Will review github.

What is left to finish would you say?

Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?

There's currently there are three places right now.

1. HFileLink
a. HFile references require include the table name as part of the filename
b. back references include table name as well
2. Snapshot names
a. In current behavior as Jesse Yates pointed out. Valid table names are valid snapshot names. Snapshots are stored on hdfs as a directory identified by their names. Though I'm inclined not to support ':' as a valid character for snapshot names so we can keep ':' as a special character?

Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name

Francis Liu Where would this happen? When would we have to have a ':', or a ',' in the fs given ns is a dir?

Presently snapshot creation will complain if it has a ':' in it

When you create a snapshot, can you use a TableName?

I think above scheme is acceptable and the best we can do right now. ns:table also fits into cf:column naming as well.

rename FullyQualifiedTableName to TableName

makes sense.

snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it

It used to follow table name semantics, but looks like it just tries on the FS and fails if its not a valid character.

A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)

Is meta public?

I've pushed some changes into github addressing the following:

• Change delimiter to ':'
• Use a filesystem delimiter ',' instead of ':' where a fqtn is referenced as part of a file/dir name
• changed filesystem table path to: <root>/data/<ns>/<table qualifier>
• updated namespace and table name valid chars. namespace now only allows [A-Za-z0-9_]
• rename FullyQualifiedTableName to TableName since all table names are fully qualified not unless explicitly stated

Some issues:

• found out that ':' is not a valid character in HDFS (scratches head). so we'll really need an fs delimiter if we stick with this delimiter.
• snapshot names character names - are all valid tables names valid snapshot names as well? Presently snapshot creation will complain if it has a ':' in it
• small and medium tests pass, didn't have time to run large yet

What do you guys think? I'll continue working on the other comments.

Check out the changes here:

https://github.com/francisliu/hbase_namespace/tree/core_8408_rebase_1

Played with this for some time. Overall, shell commands, zookeeper data, ns table, hdfs data seems to be working. Did a skim through the commits at github. Waiting for the separator change.

A possible follow up might be to hardcode .META. naming for backwards compatibility for 0.96 release? (scan '.META.' does not work right now)

Francis Liu Any status boss?

Just re-read your comments for not storing FQ name on file system. Then we can just do this for file link which already contains carefully-selected separator chars for different fields.

Yes, we can use different separator chars for fqtn depending on context. META and other places will use colon, while filesystem might use some other char (possibly comma)

@enis it looks like ':' is not in this list of not-allowed characters in ZK. I naively tried it and it works. Just curious I'm not familiar with the HBase on windows effort. They won't be running HBase on HDFS? But on a windows DFS?

@stack I believe enis is suggesting on using a different delimiter when storing FQTN as part of filenames on hdfs.

Comma is special in .META. table. Would have to adjust comparators. It would take us further away from one day being able to do memcmp on .META. rows keys.

We can use a different separator for file system then we expose to the users. Haven't checked whether zk also allows ":". Using comma for this makes sense if you ask me.

Forgot to mention, ':' is not a valid character on NTFS. So we'd have to have a different separate for full table names in HFileLinks. Possibly a switch to support this. Thoughts Enis Soztutar?

I had a chat with Stack yesterday to iron out some design details. Here's a quick summary of things:

1. We will be going with ':' as the delimiter. We'll make it so that the tableNames stored on the filesystem don't get stored fully-qualified. So that leaves HFileLinks which needs table names to be written with the namespace.

2. a TableName POJO will be the class that gets passed around where a tableName reference is needed. HTable and HBaseAdmin will have the old apis overloaded with new apis that accept the TableName pojo. In addition, the old apis will recognize fully qualified table names. This keeps symmetry between both apis. Having colon as a delimiter will guarantee there is no confusion and clear intent.

3. Since we will be using a TableName pojo everywhere internally. That means all the coprocessor hooks that have tableNames will change as well. This will break backward compatibility.

4. External interfaces REST, Thrift and CLI will not require significant changes as tables can be referenced using fully-qualified. We will have to add namespace CRUD apis to REST and Thrift at a latter point.

5. In a separate patch remove '.' prefix in non-table dirs in root dir

BTW would it help if I put the patch on RB as well? 309 files might be too much.

It's massive but not really complex. What does stabilizing involve? Reviews and all relevant unit tests passing?

I took a quick look over at https://github.com/francisliu/hbase_namespace/commit/18973af443e8c6424c978364e2d0db2a5abe1286#L5R705

Patch is massive but seems to be in the right direction. I'm thinking weeks – three to four weeks – for this to stabilize and get committed?

How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?

It depends on how we'd wanna deal with it. Presently it's addressable as all the apis recognize FQTN names with '.' as the delimiter. If we go this route there won't be much work, just some cleanup to use FullyQualifiedTableName were needed.

If we decide to have a different external and internal delimiter or overload the cli api there would be some amount of work.

In either case could be a separate patch.

How long before cli, thrift, and rest get the treatment? Could this be done later in separate patch? W/o it, it would mean that tables not in default namespace would be unaddressable?

Yeah, HC/HCM should probably be evolving at least.

On the tableOrRegionName, yeah, should probably a utility that is called to figure out which and then once that is known, the appropriate api is called.

Not sure why the commons-io. Ditto on test failures. Yes, possible unrelated to your changes.

Let me look at your work over in git.

Stack I've pushed changes into github, the changes mainly addresses one thing: make namespace as first class. I've replaced internal apis which took "byte[] tableName" as a constant to use FullyQualifiedTableName instead. For HTable and HBaseAdmin, I've overloaded such functions instead to keep backward compatibility. Also created a new branch in the repo which is a rebase off trunk yesterday. I'll address your RB comments today.

Some issues:

• The external interfaces: cli, thrift, rest and MR don't have the namespace as first class treatment yet
• I initially went by the InterfaceAudience annotations but after talking to you guys it seems this is needs some updating as HConnection and HConnectionManager are both marked public tho they shouldn't be?
• There are some apis (ie compact()) which have an overloaded parameter ie "byte[] tableOrRegionName" should we be splitting this into two apis and deprecate the old one?
• Need to create a FullyQualifiedTableName PB equivalent and update PB rpc and messages as needed
• I always needed to add commons-io as a maven dependency else things won't compile and these aren't related to my changes
• Ran small,med,large tests and have these failures which aren't related to the patch: TestHCM.testClusterStatus, TestLogRolling.testCompactionRecordDoesntBlockRolling, TestShell

Francis Liu Ping here so i can go take a looksee when you have something boss. Thanks.

Stack Getting things to compile is taking more time than I thought. Fixing unit tests now. Should hoping to have something up today. Sorry for the delay.

Francis Liu Status? I see no commits over in github?

To solve ambiguity create should have for FQ overload imo. Then, that means you always operate on existing tables if they have a dot, so you can just prevent conflicts at create time.

'/' probably won't work. Or at least, it won't work up in zk because you want to have RS watch tables come and go (or is it NS come and go). If tables, you can't set a recursive watch so would seem to rule out '/'.

Sorry, meant overload.

But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

The argument above is that the current implementation has been deployed some where so the current implementation is what we have to commit?

Can we figure out something to bring along these legacy folks Francis? They will have to restart anyways going to the new stuff? Can we treat this problem apart from how ns goes into trunk/0.95?

Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

How would that work? How would we distingush a FQTN from a plain TN if the TN has the delimiter in it as part of its name. I can see helper methods that the shell or command-line tools could use but don't see these being run on every tablename we are passed.

What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

No conflation of namespace and tablename. Clean deliniation between the ns and tablename concepts. Users can take on the new namespace feature if they want to. No hackery.

Chatting today, the '/' character came up again. Did we rule this out as a delimiter? Yeah '/' is meaningful in ext and ntfs but can't we leverage their interpretation; e.g. ns is a directory of tables? Ditto up in zk.

The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?

We are not married to the current implementation yet. But we have production users who are married to the old api and would require them to upgrade their code to use the new override apis as opposed to a table name change if we recognized FQTN strings in the existing api. I suspect other users will encounter the same problem.

Essentially what I'm asking is if it would be acceptable to recognize FQTN strings in the old api alongside implementing override?

What are we buying if we are going to support an external delimiter as well as an internal delimiter but avoid the old api from recognizing FQTN string?

*Nit side note: Shouldn't it be be called overload (not override), since we are planning on overloading a a method or am I thinking of one of the other approaches?

The legacy Y! namespacers does compllicate things. If we are not to unsettle them, the decision is made already and the override to add NS is out?

This is 0.96. Y! will have to do a 'migration' to go up on 0.96 anyways? Can you think of a 'step' that you could add for Y! cluster that would help here?

Your users are doing 'new HTable(conf, "ns.sometable");' and away they go?

But for the java apis we won't recognize an FQTN string in the public apis?

Well, there would be helper classes for making sense of the FQTN that could be used by shell, rest, etc., but, yeah, the thought would be that in the API, there is no namepacing, not unless you ask for it explicitly.

Let me look at the trick for how much we'd have to carry into the future.

You mean Y! They are using '.' for separator currently?

The only dots we use are for namespaces and that's only in our sandbox so no worries there. My concern was that we'd have to ask users to upgrade their code in order to start using namespaces.

So we'll have an internal delimiter '.' and an external one for external tools such as cli, rest, etc. But for the java apis we won't recognize an FQTN string in the public apis?

I thought you had trick to make '.' work?

Yes, I do. Enis Soztutar pointed out that we will have to live with that trick in later versions. Just making sure you guys are ok with that. Have a look at it on github, just look at TableName.java that should help you get the gist of it.

One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS.

You mean Y! They are using '.' for separator currently?

As opposed to having just a table name then it's just a simple config change and restart.

No override of APIs?

We have a special KV comparator for .META. table. It leverages the ',' position as delimiter between table name and startcode in region name. I suppose we could change the KV comparator so it looks for ','.

I thought you had trick to make '.' work?

ZK restrictions:

http://zookeeper.apache.org/doc/r3.4.3/zookeeperProgrammers.html#ch_zkDataModel

If only they printed the chars instead of code points.

We can broaden the range if we encode the delimiter when stored on fs and zk. We are mainly limited by region and table names.

+thrift as well. That is why I am in favor of option 2 now.

',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?

I think we can use this. We are relying on it for partitioning region names, and in META. So it should be compatible with that framework.

Also I'm wondering how this will affect the rest service. Will we need a v2 set of apis to support this?

The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

I see, so what delimiter are we planning on using? '^' and '@' isn't appealing. ',' looks nice but you mentioned it's already used any way we can update that cleanly to take this into account?

I think the old API will be against default NS.
The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

Is it possible to support FQTN as well? Or is going this route to avoid surprise completely? One concern that I see internally is that all our users will have to update their code to use the new apis so we can migrate them to new NS. As opposed to having just a table name then it's just a simple config change and restart.

If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

I think the old API will be against default NS.

The FQTN (Fully Qualified Table Name) would be an internal or something that could be passed to external tools (command-line, shell).

One problem with option 4 is that we want to pay the price of migration only one between 0.94->0.96. If we do that, then it means we have to carry the exception tables code for all the releases going forward. Option 1 better than this I think? Note that surprise #1 also applies here as well.

Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

It should use the default ns. I think the idea is that there will not be a public facing thing called "fully qualified table name" in Elliot's approach. Although internally, we will need one, hence my tendency to go with option 2 over 3 (see my above comment): "namespace,table" seems good enough for me.

Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

The approach I proposed earlier would avoid having to do all the api stuff as part of the first namespace checkin as well as make use of '.' as a delimeter. The suprises are as I mentioned. We can incrementally add the apis.

Sounds like we are going with overloading all the existing apis to take a namespace parameter. If so what would be the behavior when using the old api? Will it always reference default namespace or will we support fully qualified table names?

For some reason I'm not getting any jira or RB emails. Will take a look.

Francis Liu There will be a migration evaluation script that will looks for presence of stuff like hfile v1s – they must be compacted away before you can upgrade – and this same step could check table names and if any w/ dot found, list options. This script would be run against 0.94 install before shutting down for upgrade (Yes two scripts, a checker, and then a doer).

Francis, we should still do the Elliott suggestion even if dot, right? The dot would be for 'external' tools or a useful facility in shell but we want namespaces to be first class in API too.

Did you get my review comments up on rb francis?

On dots in namespace, no, if it simplifies.

exceptionNS.add(tableName.getNamespaceAsString()); 

What is the current thinking on dots in namespaces and names? Presumably one table could prevent the creation of multiple namespaces if dots are allowed in namespace name, which I thought they are

Oops sorry I guess I'm talking about two scripts. One to check if some surprising migration needs to be done and provide links/options. And another that does the actual migration.

I thought of a way to implement Sergey Shelukhin idea. It was simple enough so I thought I'd give it a try. Essentially keep an in-memory list of tables which make use of the delimiter (ie '.') and consider these tables as exceptions to the namespace rule and handle them properly to make sure they are part of the default namespace. Have an added constraint that prevent creation of namespaces and tables that would conflict with any of the exception tables (ie ns1 and ns1.foo).

Surprise here is:

• you can't create tables with the delimiter no longer unless you create the appropriate namespace.
• you can't create tables/namespace which conflict the exception tables/namespaces
• the exception list is derived by scanning the default namespace directories in .tmp, .data and .archive

Here's a sample of how it works. I've updated the TestNamespaceUpgrade test to verify that it works:
https://github.com/francisliu/hbase_namespace/tree/core_8408_exception_list

Stack, I'm leaning towards having the migration operation done manually by calling a script as well. Which options do we provide the user? Also it might be better if the script is portable enough that they can run on an existing 0.94 cluster so they don't have to find out during the actual upgrade process.

I was thinking more along the lines of doing the non-surprising thing silently (if the user doesn't want to use namespaces he will never notice anything at all), and allowing the user to resolve conflicts as necessary while preventing them from unwittingly shooting himself in the foot; but migration-time options also work.

Option 4 would get us the '.' back (that would mean that we'd do options 3 and 4).

Migrating, we will have a pre-update step where you run a script and it will check your install for v1 hfiles AND if you have tables with dots in them. If you have dots in your table name, you will be pointed at a page that describes the options available to you.

Option 4 is to use a dot, not create any automatic namespaces, have well-defined name resolution rules, and prohibit creating conflicts under those (also prohibit any further dots in table names for good measure). E.g. if user has org.my.cool.table, org.my.foo.table they go under default, and namespaces org, org.my, org.my.cool, and org.my.cool2 cannot be created at all until this table is renamed (or maybe they can be created by explicitly specifying that all tables should be renamed/moved under namespace keeping essentially the same name given the dots).
That way it's explicit to the user what needs to be done (no shooting-in-the-foot); there are dots; and there's minimum surprise.
The extra dev work is for mapping such table names to HDFS paths/etc.

So, that leaves '^', the acute or hat character: namespace^table?

Option 3 seems to involve less technical debt (or chance of tech debt).

If we use '@', won't we have to swap the order so it is table 'at' namespace? (Or, the '@' would seem to imply that).

Between 2 and 3, if we choose 3, can't we keep more hidden from the user – or allowing your 'external tools' argument, we should do 2 and 3?

BTW regarding separator chars:
"\/:*?<>|" NTFS does not allow these chars
"\/~" Meaningful in ext/ntfs.
"-" and "_" cannot be used because they are legal in table names.
"#" Comment in many SQL dialects, not compatible with URL encoding.
"$%&" Used in shell scripts

This leaves: "!@^+=," if we disregard parenthesis chars. These also seem to work for HDFS. "," is our internal separator for region names and META naming. @ also seems good enough. Haven't tried these for znodes.

It seems that we could not reach an agreement on which proposal to implement, where they are
1) use dot as separator, autogenerate namespaces on migration
2) use one of the candidate char's above as ns and table separator.
3) In all places that require a table name, override with accepting namespace.

However, it seems that we definitely have to have an internal separator char and a deterministic way of encoding <ns,table> pair into a string and decoding it back. This is because we need to do this for znodes in zktable, tablelockmanager, etc as well as META. So, I think it is a matter of whether we want to expose the separator char between proposal 2 and 3. I am in favor of doing 2, since it will allow better interop with external tools, and the code changes needed for 3 is significant.

Francis Liu Are we going to implement the Elliott suggestion adding overrides for namespaces? I was chatting w/ Nick Dimiduk last weds and he made a comment that helped: he said that if a choice between devs doing more work to save users 'surprise', then the answer is clear.

Another thought is that we need to hurry up and get this in. I'll do a review now. Who else is reviewing?

Thanks Francis.

As discussed I've created a github repo please see HBASE-8408. Please give it a try and let me know what you guys think.

Stack It's past testing and we're baking it in our sandbox environment where we have active users. No issues so far.

I am doing the review, but could definitely use more reviews. After the first patch (HBASE-8408) is +1'ed I will create a branch and commit there, since we will need the rest of the subtasks in shape before proposing the merge.

What is state of this feature? What else needs to be done other than more review and test? Are we going to make a branch for it?

Francis Liu How much testing of this feature has there been? Thanks boss.

Ran tests see below. "testDelayedRpcImmediateReturnValue" seems to be flakey. It passed when I reran the class separately.

Small and medium (and large):

Results :

Tests in error:
testClusterStatus(org.apache.hadoop.hbase.client.TestHCM): Unexpected exception, expected<org.apache.hadoop.hbase.exceptions.RegionServerStoppedException> but was<junit.framework.AssertionFailedError>
testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details.
testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Index: 1, Size: 1

Tests run: 1494, Failures: 0, Errors: 3, Skipped: 13

Large Tests:

Results :

Tests in error:
testRunShellTests(org.apache.hadoop.hbase.client.TestShell): (RuntimeError) Shell unit tests failed. Check output file for details.

Tests run: 512, Failures: 0, Errors: 1, Skipped: 11

I believe the necessary code to verify this migration approach is in place. Please let me know if there are any concerns.

I've uploaded a first stab of the migration code on RB and HBASE-8408. Haven't addressed any of the comments yet so I could get this out first.

I added a unit test which tests migration on a tarballed 0.94 rootdir. It does simple verification that the tables and snapshots have been migrated properly.

It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

No it shouldn't. There was no consensus on the list and this is an exploratory effort so no need of constraint.

One thought I had was that if all tables need to be moved, mayhaps we just say snapshots are broke on upgrade or rather than a tool for table rename, instead, we'd have a tool that did surgery on snapshots to adjust them to match the new layout (advantage of this latter is that it can be done offline, not inline with migration – though, I have no idea this would work... ask the snapshotters Jon, Matteo and Jesse).

a RenameTable tool will be provided to rename offline tables

It should be noted that the rename is done by first taking snapshot of the table and cloning to the new table name.

It seems we've come to a consensus on the upgrade approach.

In essence it'll do the following:

• Tables without '.' will be moved into the default namespace.
• Tables with '.' will be move into new namespaces
• namespaces will be delimited by the last '.' in the table name
  ie org.apache.hbase.MyTable, namespace = org.apache.hbase
• In both cases the oldTableName is the same as the fullTableName
• all existing apis and cli commands will be expecting full table names unless explicitly stated
• a RenameTable tool will be provided to rename offline tables

So I'll start implementing it.

Discussion is here:
http://grokbase.com/p/hbase/dev/13598jegtp/discuss-namespace-delimiter

Enis Soztutar do you have any bandwidth? Since you've shown interested earlier

I am up for it.

Would a quick meetup with the people involved help iron things out?

+1. It is great that 0.94 version is in shape. The remaining questions are to handle migration, and the separator issue, and testing I guess.

Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

I am not a big fan of svn branches, but if it will help in reviews and testing, I can help with that.

Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

Our 0.94 patch is almost fully-baked so far so good...we already have it running in sandbox with users. Would a quick meetup with the people involved help iron things out?

RB patch is attached to subtask.

Looking at what is up in rb, it is a massive patch that still has the migration part to be done. My fear is that we'll race to get this into 0.95 and it will not be well baked enough – and it can destabilize 'normal' operation. I am currently inclining against its inclusion (open to convincing otherwise but getting my opinion out there).

NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

In patch attached here. May be gone in the rb version.

This can be done later, right? You'd have to do it in hbase.

Yep

Are we saying the default ns is called 'default'? Or that it has no name?

It has no name similar to the java default package.

Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)

I wish too. We can refactor after this to create one?

How much of this design has to be in place for 0.96 to ship?

We're watching out for backwards compatibility? I believe core namespace should be enough. We can put the rest in 96.1, quota stuff needs addition to the cli. Security has some changes as well to support namespaces but it's just an extension.

Below is interesting (smile)

It's our secret version

Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).

My line of thinking was that namespace is a component of tableName ie "tableName = <table namespace>.<table qualifier>". Just like column = CF:CQ.

NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

Where did you see that? Group information is added as a KV pair to the NamespaceDesc.

NamespaceDescriptor implements java Serializable

Oops will remove that

Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

Enis Soztutar do you have any bandwidth? Since you've shown interested earlier

IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

Looking at the comments there aren't any major issues apart from the delimiter. The namespace affected tests are passing.

Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

Either sounds fine to me.

Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

Working on my existing patch we can try out elliot's suggestion which would essentially require the new interfaces

If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

Yep it's possible. Though the patch will need some small changes in assignment manager and LoadBalancer interface, etc. Better to get those in soon if we're in a hurry. I can separate those out as a separate patch as well?

Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

It's gonna be hard, will give this a try.

Regards sponsoring this issue, would suggest more than Ted Yu as volunteer is needed if serious about getting this in to 0.95/0.96 (I am working on 0.95 testing and last issues so do not have bandwidth to help here)

IMO, the time for new features in 0.95 is loooooonnnnnnggggggg past so hard to justify holding up release for this.

Lets get a feature branch going. I can help set up apache build of the branch or do it over on andrew's ec2... so can get some test runs going.

Need to work out the dot problem. Can we do the search in two places as is suggested over on the list (internally, Elliott has suggested how to implement); i.e. only new tables go into namespace? (Cannot gauge the migration problem till know how this plays out)

If this goes into 0.95, table grouping can be done after 0.95 as CPs? Not clear after reading above.

+1 on getting rid of funky system tables stuff (but yeah, as per Francis above, they were given the funny names so they sorted before user tables).

Breaking patch into pieces will help yes, especially ones that collect into one place our interaction with filesystem (see Matteo's recent work).

On the design, needs date, author, and JIRA number.

As a first step we only intend to limit the number of tables and regions a given namespace may contain.

This can be done later, right? You'd have to do it in hbase.

Are we saying the default ns is called 'default'? Or that it has no name?

NamespaceController coprocessor running on region servers to correctly enforce quota restrictions

Is this so we can get callbacks on change? (I wish we had in place a general config. change notification mechanism that this could leverage rather than have to build its own)

How much of this design has to be in place for 0.96 to ship?

<ROOT>/data/<NAMESPACE>/<TABLE>

Why the need for 'data' in above?

Looking quickly at the attached patch (will look at rb later):

Below is interesting (smile)

-        <hadoop.version>0.23.3</hadoop.version>
+        <hadoop.version>0.23.7-SNAPSHOT</hadoop.version>

Patch introduces TableName and TableName is used to hold namespace and the name of the table. NS is a concept superior to tables. It should be elsewhere, in a NS object? (model seems off).

NamespaceDescriptor implements java Serializable?

NamespaceDesc.. has notion of groups. Is that right given they are add on for later?

Will look at rb patch next. What is attached doesn't seem enough. Will be back.

Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

Essentially we need to keep the "HTable(byte[] tableName, Configuration config)" constructor. The current implementation would assume a table is part of the default namespace if only the table name is provided.

Apart from potentially confusing our users. Do you have any other concerns against supporting a fully-qualified table name?

Should we support cross-namespace constructs ? If so, namespace would be exposed to user.

From my point of view the most important usability concern should be the java api, with thrift second. My proposal would allow cross namespace very easily by just creating two different HTable's in different namespaces.

The shell is nice but it's just a convenience utility. We shouldn't make the most used case harder or more confusing chasing some dream of having a sql shell. In my opinion since HBase is not Oracle's 11g, it shouldn't value being like their sql.

We need to expose it mainly for backwards compatibility.

Can you talk a little about this. I thought the examples I posted showed we could be backwards compatibile by adding an optional paramter (htable would be treated in a very similar manner).

I see. We need to expose it mainly for backwards compatibility. Also isn't being able to reference a fully-qualified table name common practice in databases? So users aren't learning something new here.

Should we support cross-namespace constructs ? If so, namespace would be exposed to user.

For Oracle, some Namespaces are reserved:
http://docs.oracle.com/cd/B28359_01/appdev.111/b31231/appb.htm#BABFFDFC

This means an ordinary user query can easily reference more than one namespace.

At the very least we will have to store table names fully qualified in the catalog tables.

True, but we don't need to use . here, or expose it to the user.
So I'm saying most places should use a different character to store the table name, and I would argue that we shouldn't expose that fully qualified table name to the user at all.

With a namespace

create 'namespace', 't1', 'f1'

Places something in meta like: 'namespace,t1,,hash'

Put Usage:

put 'namespace', 't1', 'r1', 'c1', 'value'

or

use 'namespace'
put 't1', 'r1', 'c1', 'value'

default space

create 't1', 'f1'

places something in meta like: 'default,t1,,hash'

put 't1', 'r1', 'c1', 'value'

or

put 'default', 't1', 'r1', 'c1', 'value'

Comma is already reserved and we can use it for everywhere that we need a string name. But we shouldn't expose that to the user. We shouldn't overload table name with two different parts. It gets confusing for users when one parameter actually represents two different things (namespace and table name).

Elliott Clark We can add that an the interface level to support both means of identifying tables. At the very least we will have to store table names fully qualified in the catalog tables. Storing it on hdfs and zk the same way makes things easier as well.

By default namespace did you mean something like a "use <namespace>" command in the cli?

I think that adding a parameter for namespace is cleaner. It would mean that we don't add need to rename tables. People can continue to keep their table names. We also then get an easier path to infer the default namespace.

Another option is to factor the helper piece out of this class and get that committed quicker. Apart from extra work, we won't be able to verify everything is covered without the rest of the patch. Thoughts?

Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

stack Would we be able to get it in quicker with a feature branch? The reason I am asking is because part of what's challenging about this enhancement is catching all the code which makes assumption on path changes. And that will continue to grow until we provide the helper functions to be used. We have an 0.94 version of this patch working in sandbox with active users. So I believe basic functionality is working. The big thing missing is the migration script (We don't use '.' in table names).

That's right.

Maybe you can start discussion on dev@hbase for this topic.

Ted Yu thanks but there doesn't seem to be any consensus?

@Francis:
Take a look at the summary here:
http://search-hadoop.com/m/K0wzj1V5vpu

Delimiter and directory structure for namespace support is discussed in the middle of the above summary.

For security, should each namespace have its own permission settings ?

Yes, the permissions are similar to to directory privileges. You would need write access to a namespace to be able to create and drop tables. We already have a patch for this in 0.94. We'll put it up once we sort out the core patch.

For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

yep see subtask. We also have this implemented will put up patch once most things are sorted out with core patch.

@Francis:
Can you add new tests for this feature ?
The new tests should include unit tests and, integration tests.

There are unit tests I can have it run as an IT test as well. Though the real test really is the existing tests as the main issues are updating/replacing code segments which make assumptions about the directory layout. The tricky ones were in snapshots, hbck and splits. In this patch I've gotten all of those to pass. We can add more tests to these components as needed.

Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?

When I was asking about a different delimiter. No one was really happy with anything else but a ".". If you have a suggestion please let us know.

I volunteer to be the sponsor.

Never mind merit, this issue has it in spades, if this radical refactor is to make it into 0.95, the activity needs to pick up. For integration to happen, the patch needs to be freighted w/ tests and migration code able to promptly move hbase from its current layout to namespace layout. The patch when done should have a glow of confidence w/ ready answers for how to deal with issues such as pre-existing tables that have a '.' in their name.

It has been suggested that we make a branch to speed dev on this issue which should help move things along; a branch will need a sponsor/conmiitter. I could volunteer but there are probably better mentors than sloppy I who can help along this issue and advise on what is needed to make the cut-off date fast looming.

@Francis:
Can you add new tests for this feature ?
The new tests should include unit tests and, integration tests.

Using dot ('.') as separator is intuitive. Would using some other character make sense so that migration effort is lower ?

For security, should each namespace have its own permission settings ?

For enforcing namespace quota, that would be implemented in a follow-on JIRA, right ?

Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?

That's disk space quota. Right now we plan to enforce it using hdfs quota and see how that goes. That way we don't have to reinvent the wheel.

What about naming the new command create_namespace (with verb at the beginning) ?

I wanted to namespace the commands . Let me know if the convention is different for HBase.

Going over the design doc, I have a few questions:

As a first step we only intend to limit the number of tables and regions a given namespace may contain.

Regions may have different sizes across namespaces. Would it make sense to limit the (estimated) total size of regions in a particular namespace ?

namespace_create ‘<namespace>’, {

What about naming the new command create_namespace (with verb at the beginning) ?

Thanks Francis, I'll take a look at this today.

I've updated the design doc with a component diagram and some notes to make reviewing the code a bit easier. Let me know.

Enis Soztutar I would very much like to get it in as well. I've addressed your comments. Also I've broken down the pieces into subtasks. The patch on RB is for the first subtask. We currently have 1-3 running on 0.94. We'll forward port the 2nd and 3rd piece as soon as the first task completed to keep things manageable.

Francis Liu, I would like to get namespaces in 0.96, because it fits well into the singularity. Getting it in later would be hard. I've put up some comments at RB. I can offer some help should you need with review / split / etc.

I plan to revisit this test after finishing one round of review.

Without the patch, I got:

Running org.apache.hadoop.hbase.master.TestMasterFailover
2013-03-28 16:57:21.814 java[4597:db03] Unable to load realm info from SCDynamicStore
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec

Oh, maybe it's just my machine then. What was the purpose of your previous comment then?

Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

AFAIK, if the META table goes offline due to RS failure. There is no special case to make it's assignment a priority. And priority is based on lexical ordering, in which case we will need the funky prefix. Not unless we'd want to add complexity to handle that logic.

BTW the check is not in yet. But I'd also like to remove '.' as an option for table name. Beyond the proposed namespace delimiter usage.

Without the patch, I got:

Running org.apache.hadoop.hbase.master.TestMasterFailover
2013-03-28 16:57:21.814 java[4597:db03] Unable to load realm info from SCDynamicStore
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.262 sec

TestMasterFailover failed with patch. In test output, I saw:

Yep, as I mentioned in reviewboard. There are 3 tests that are failing. Have yet to determine if it is related to this patch.

I am still going through your code.
Would user be able to create table in default namespace after migration ?
Put in another way: what privilege is required to do the above ?

namespace affiliation is determine by table name during creation.

ie

#create table in namespace foo
create 'foo.my_table','f'

#create table in namespace default
create 'my_table','f'

TestMasterFailover failed with patch. In test output, I saw:

    <error message="test timed out after 180000 milliseconds" type="java.lang.Exception">java.lang.Exception: test timed out after 180000 milliseconds
  at java.lang.Thread.sleep(Native Method)
  at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:133)
  at org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:469)
  at org.apache.hadoop.hbase.HBaseCluster.waitForActiveAndReadyMaster(HBaseCluster.java:206)
  at org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:759)
...
2013-03-28 16:27:21,666 INFO  [Master:0;10.11.2.252,51376,1364513236949] master.HMaster(954): Failed to contact server
java.io.IOException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
  at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
  at com.sun.proxy.$Proxy17.getRegionInfo(Unknown Source)
  at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRegionInfo(ProtobufUtil.java:1269)
  at org.apache.hadoop.hbase.master.HMaster.assignSystemTables(HMaster.java:949)
  at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:780)
  at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:536)
  at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: java.io.IOException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Server is not handling protocol org.apache.hadoop.hbase.client.AdminProtocol
  at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:166)
  at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1871)

  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1321)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
  ... 6 more
...
2013-03-28 16:30:39,937 INFO  [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.RegionStates(264): Region {NAME => '-hbase-.-meta-,,1', STARTKEY => '', ENDKEY => '', ENCODED => 715583861,} transitioned from {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082} to {-hbase-.-meta-,,1.715583861 state=PENDING_OPEN, ts=1364513439937, server=10.11.2.252,51480,1364513407082}
2013-03-28 16:30:39,938 WARN  [MASTER_META_SERVER_OPERATIONS-10.11.2.252,51577,1364513413623-0] master.AssignmentManager(1869): Failed assignment of -hbase-.-meta-,,1.715583861 to 10.11.2.252,51480,1364513407082, trying to assign elsewhere instead; try=10 of 10
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is in the failed servers list: /10.11.2.252:51480
  at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:785)
  at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1414)
  at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1299)
  at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
  at com.sun.proxy.$Proxy17.openRegion(Unknown Source)
  at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:603)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1796)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1431)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1410)
  at org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(AssignmentManager.java:2303)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:100)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:124)
  at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:69)
  at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:130)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed.

I am still going through your code.
Would user be able to create table in default namespace after migration ?
Put in another way: what privilege is required to do the above ?

I've renamed the meta table to become -hbase.meta

Nice. This is the same as what was proposed over in HBASE-8093

Yes, let's solve this issue here once and for all. thinking about it, I have a hard time understanding why we need to use special chars in catalog tables. Can't we just have a namespace called simple "system", and have tables named "system.META" and "system.ROOT". I say no funky -, _, . escaping stuff.

Looks like the latest patch wasn't properly generated.
In HMaster.java:

Sorry about that. Updated the patch.

This of default namespace akin to the default package in java.

Thanks for reviewing ted.

What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.

It's for existing tables that never used '.' as part of the table name. So they don't have to be renamed. They will all fall into the default namespace bucket and be accessed by the application the same way.

Looks like the latest patch wasn't properly generated.
In HMaster.java:

+<<<<<<< HEAD
+=======
+import org.apache.hadoop.hbase.NamespaceDescriptor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ServerCallable;
+import org.apache.hadoop.hbase.exceptions.ConstraintException;
+import org.apache.hadoop.hbase.exceptions.DeserializationException;
+>>>>>>> first forward port of namespace trunk patch

I am beginning to go over the code on review board.

-there's a special case with tables in the default namespace: <table qualifier>

What tables would be placed in default namespace ? I think this (default namespace) may introduce confusion if there is no concrete use case.

I've renamed the meta table to become -hbase.meta

Nice. This is the same as what was proposed over in HBASE-8093

Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
I find the latter more intuitive.

If you find that agreeable. We can keep it that way.

Would the group table from HBASE­6721 belong to the above namespace ?

Yep. depending on which ends up going in first will have to make that change.

Sorry for the delay I was OOO last week. The patch is way bigger than I anticipated changing the path hierarchy took bulk of it as assumptions about it was all over the code. Here's an initial draft of the patch, functionally it's complete. I need some feedback before I can go further.

https://reviews.apache.org/r/10167/diff/#index_header

Some questions on design doc.

multiple namespaces can be part of the same region server group.

Can you elaborate a little bit ? From comment above: Internally we plan to do it using regionserver groups (one namespace per group).
I find the latter more intuitive.

hbase ­ admin tables:­ROOT­, .META., ACLs

Would the group table from HBASE­6721 belong to the above namespace ?

How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible.

Maybe we should have WALs per namespace per server . If we want other namespace based features (ie replication) going this route makes sense?

Internally we plan to do it using regionserver groups (one namespace per group). Since we don't want our tenants contending for other resources as well (disk, mem, cpu, etc).

Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

It'd be good if we can leverage hdfs quota since it's already there and has worked well (at least for us). That's also less complexity for us, then we can focus on adding HBase specific Quota (ie regions, tables, etc).

I've also changed the tableDir on hdfs to include namespace as the parent directory

How do you plan to enforce this? WAL's completely bypass the quota, and if a ns's quota fills up, it cannot flush, and compact, which will affect all of the other namespaces. I am not against the idea of having a parent namespace dir for tables, but just wondering whether this is really feasible. Can't we enforce quotas at the HBase level, since we are doing ACL's at the application layer for the same reasons.

So it seems we're fine providing a migration script. So I've gone ahead and produced a rough patch with the change.

The general idea is to have a TableName class which encapsulates both namespace and tableQualifier (AKA the old table name). And this will be what gets passed around as much as possible internally. Table names are referenced fully qualified otherwise.
I've also changed the tableDir on hdfs to include namespace as the parent directory. This will enable us to set HDFS quotas by namespace.

I skipped the CRUD related stuff since that seems pretty straightforward. Let me know if this approach is acceptable.

@Francis:
Can you work on the trunk patch first ?
If it gets ready before 0.95 RC0, it can go into 0.95

Thanks

Yeah, that would be great! Thanks Francis

Jesse Yates I'm hoping to get an dev drop (0.94) by end of next week. I'll try to squeeze in a trunk patch. Does that work for you?

HBASE-7999 has 0.96 as Fix Version.
If we agree that this feature is superset of HBASE-7999, we should focus on this feature.

Well, there will be an upgrade script... then this will have to be added to it.
Also, trunk is now 98 so technically this should go into 95 then

Francis Liu what kind of timeline are you planning for doing this work? I love to see it subsume HBASE-7999, but this is a much more involved change than what's going on there.

my point about backward compat was about the storage

IIRC we don't support backward compat for 0.96 backwards. As long as this goes into trunk, it should be fine going forward.

The latter may not necessarily be backward compatible...

Can you give an example to this? Thinking about it more, it seems to me you'll just end up with a lot of namespaces?

I see, I am assuming we have to store namespace as part of the region name. And store them fully-qualified on hdfs/zookeeper/etc. Else we would be forced to have all table names to be globally unique which would different from database semantics.

Another concern is if a user can run the same application code against 0.96.

ie if I wanted to scan:

> scan 'foo.bar'

Pre-NS this would scan table 'foo.bar'. Post-NS, the system would parse this out as table bar in namespace foo. One way we could deal with this is read the table as Post-NS if it doesn't access read it as Pre-NS, check again if not then fail.

my point about backward compat was about the storage, not display. I.e. if people want to create a table "a.b" without namespace and be confused while viewing some table list, I think it's ok. Maybe we can disallow tables with dots.
If they have existing table "a.b" and NS is stored separately, again the worst thing that happens is user gets confused, reads the release notes and renames it. Should be ok to.
But if we /store/ NS in table name, then with table a.b, without special handing, system gets confused (thinks it's in NS "a") after upgrade

Hmm, ns.table is definitely more intuitive than /. SQL also uses that convention.

Good point. For now I'll stick with '/' if there are no objections. And revisit '.' if anyone feels strongly about it.

The latter may not necessarily be backward compatible...

Given that '.' is already used. Should we pick another delimiter for namespaces? Or should we provide a backward compatible way to support this. Ie creating namespaces for tablenames with '.'?

I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

They will be first class citizens. There will be a namespace table for namespace meta information. Also embedding namespace information in table name does not prevent this, we are just introducing the notion of fully-qualified table names (which we should introduce anyway). And store them in this form in meta/root.

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

I think we should have namespaces as first class citizens. Namespaces have been traditionally used for grouping tables, setup replication, restricting access, etc. As a database, we can also use namespaces for acl, repication, backup, etc.

w.r.t. dot in table name:

public static final String VALID_USER_TABLE_REGEX = "(?:[a-zA-Z_0-9][a-zA-Z_0-9.-]*)";

So dot should be allowed

Thanks for the clarification. It seems like '.' and '-' isn't allowed only if it's the first character. For backward compatibility why don't we create namespaces for those tables that are named that way?

HBASE-7999 provides System table support.

@Francis:
In the future when you want to correct your previous comment, you can quote the previous one and provide correction below the quote.

reposting coz of typos:

+1 on making core (as opposed to cp-s I assume?)

Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

"Given the level of autonomy namespaces will provide tenants. " what does this mean?

In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?

Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

w.r.t. dot in table name:

  public static final String VALID_USER_TABLE_REGEX = "(?:[a-zA-Z_0-9][a-zA-Z_0-9.-]*)";

So dot should be allowed.

+1 on having namespaces in core. Namespace/database's are universally understood in terms of the database space. We can keep the grouping of regionservers per namespace out of core, and deliver that as a part of the other region grouping issue.

+1 on making core (as opposed to cp-s I assume?)

Yes. Making this core would mean I'd have to break the task into cp and core. CP - for region server groups integration and quota control. Core - for basic namespace functionality.

"Given the level of autonomy namespaces will provide tenants. " what does this mean?

In a security-enabled cluster only system admins can create table, namespaces will introduce the notion of namespaces admins which will be granted to tenants. Thus enabling them to create tables themselves.

Having it part of the table name makes the changes less invasive (changes in meta schema, HTable apis, etc). Though I agree it would be nice to make this

Last I checked '.' around allowed as part of the tablename. The cli will bork if '.' is used?

As part of backward compatibility we can skip renaming root and meta and just explicitly support that they are part of the system namespace? These tables are already treated differently anyway?

+1 on making core (as opposed to cp-s I assume?)
"Given the level of autonomy namespaces will provide tenants. " what does this mean?
What kind of resource allocation will be possible, except for server groups (just examples, to understand it better)?

Then, about storing, I suggest not making it part of table name. It seems brittle, and will limit our options if we want to add features lately due to backward compat.
Also, how do we handle existing backward compat if someone already has a table name "foo.bar"?
Will root/meta have to be renamed hbase.root/hbase.meta to be in correct namespace, and how will it affect current assumptions about sorting if yes?

Initial draft of design. This originally was intended to be implemented as coprocessors thus it's design was made to be as non-invasive as possible.

Enis Soztutar Suggested that it would be better to make this part of core. I'd be up for doing that and open to other changes to make things more integrated.