[FLINK-12771] Support ConnectorCatalogTable in HiveCatalog

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: Bowen Li
• Reporter: Bowen Li
• Created: 06/Jun/19 21:33
• Updated: 16/Oct/19 15:42
• Resolved: 16/Oct/19 15:42

Description

Currently HiveCatalog does not support ConnectorCatalogTable. There's a major drawback on this when it comes to real use cases, that is when Table API users set a HiveCatalog as their default catalog (which is very likely), they cannot create or use any inline table sources/sinks with their default catalog any more. It's really inconvenient for Table API users to use Flink for exploration, experiment, and production.

There are several workaround in this case. E.g. users have to switch their default catalog, but that misses our original intention of having a default HiveCatalog; or users can register their inline source/sinks to Flink's default catalog which is a in memory catalog, but that not only require users to type full path of a table but also requires users to be aware of the Flink's default catalog, default db, and their names. In short, none of the workaround seems to be reasonable and user friendly.

From another perspective, Hive has the concept of temporary tables that are stored in memory of Hive metastore client and are removed when client is shut down. In Flink, ConnectorCatalogTable can be seen as a type of session-based temporary table, and HiveCatalog (potentially any catalog implementations) can store it in memory. By introducing the concept of temp table, we could greatly eliminate frictions for users and raise their experience and productivity.

Thus, we propose adding a simple in memory map for ConnectorCatalogTable in HiveCatalog to allow users create and use inline source/sink when their default catalog is a HiveCatalog.

Issue Links

• links to: GitHub Pull Request #8720