A table T1's primary cluster is C1 meaning :1) C1 contains all data that is available in all other clusters.
Does this mean that if T1's primary cluster is C1, then all of the partitions in T1 must also have have their primary partition set to C1? If that's the case then primary cluster should probably be a table level property, and the list of replica clusters can be a table/partition level property.
2) write is only allowed in this cluster for table C1. but need to allow exceptions here
What are the exceptions?
4) all data changes to T1 happened in the primary cluster should be replicated to other clusters if there are any secondary clusters. but there should be a conf to disable it as there are some exception situations.
What are the exceptions?
How will dynamic partitions work? Where will new partitions get the list of replica clusters from? Will they inherit it from the table definition?
Hive now supports insert-append into a partition (
HIVE-306). Suppose that the metadata for a particular partition indicates that it is replicated to clusters C2 and C3. If I insert new data into the partition in the primary cluster C1, then the metadata is now invalid. How is this going to be handled?
2) add new interfaces which do exactly the same set of functionalities as old ones but using a different name (use _on_cluster suffix maybe?) and have a cluster parameter
This is going to introduce new codepaths that need to be tested separately, and also double the amount of work people need to do every time a new metastore API call is created. I don't think this is a good approach.
3) overwrite database name for the purpose of cluster name. And allow a table co-exist in multiple databases. But that require to promote table to top level citizen, and degrade database. For example, "show tables" used to scan all tables in current db, but now need to scan all tables in all databases.
I don't think this is an option since it breaks backwards compatibility and effectively changes the whole notion of what a db/schema is. A lot of people in the community already depend on this feature.
1) add a cluster parameter to existing thrift interfaces
This sounds like the best option to me. I think Thrift supports API evolution via default values for missing parameters, but setting a default value in this case may be a little tricky.
Also, instead of modifying the Thrift interface, is it possible that you could instead leverage the work that's being done in HIVE-2720?