Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
Imagine a scenario when we enable namespaces for phoenix on the server side and set the property phoenix.schema.isNamespaceMappingEnabled to true. A bunch of clients are trying to connect to this cluster. All of these clients have
phoenix.schema.isNamespaceMappingEnabled to true, however
for some of them phoenix.schema.isNamespaceMappingEnabled is set to false and it is true for others. (A typical case for rolling upgrade.)
The first client with phoenix.schema.mapSystemTablesToNamespace true will acquire lock in SYSMUTEX and migrate the system tables. As soon as this happens, all the other clients will start failing.
There are two scenarios here.
1. A new client trying to connect to server without this property set
This will fail since the ConnectionQueryServicesImpl checks if SYSCAT is namespace mapped or not, If there is a mismatch, it throws an exception, thus the client doesn't get any connection.
2. Clients already connected to cluster but don't have this property set
This will fail because every query calls the endpoint coprocessor on SYSCAT to determine the PTable of the query table and the physical HBase table name is resolved based on the properties. Thus, we try to call the method on SYSCAT instead of SYS:CAT and it results in a TableNotFoundException.
This JIRA is to discuss about the potential ways in which we can handle this issue.
Some ideas around this after discussing with twdsilva@gmail.com:
1. Build retry logic around the code that works with SYSTEM tables (coprocessor calls etc.) Try with SYSCAT and if it fails, try with SYS:CAT
Cons: Difficult to maintain and code scattered all over.
2. Use SchemaUtil.getPhyscialTableName method to return the table name that actually exists. (Only for SYSTEM tables)
Call admin.tableExists to determine if SYSCAT or SYS:CAT exists and return that name. The client properties get ignored on this one.
Cons: Expensive call every time, since this method is always called several times.