[IMPALA-9670] Fix unloaded views are shown as tables for GET_TABLES requests

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Component/s: Catalog
• Epic Link: fine-grained metadata in catalogd
• Epic Color: ghx-label-12
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 17/Apr/20 13:17
• Updated: 09/May/23 02:00
• Resolved: 24/Jun/22 13:09

Description

JDBC/ODBC clients will send GET_TABLES requests to get the list of tables of a given db. Table types and comments should be returned as well. However, for unloaded views which are represended as IncompleteTable, Impala doesn't know the exact table types and return them as tables by default. This causes some troubles for applications that needs to distinguish views.

Problems to fix in this JIRA:

• Views are shown as tables when Impala launches (all views are unloaded).
• Views are shown as tables after creation (the created view is unloaded).

Possible solution is adding table type to IncompleteTable:

• IncompleteTable s are created in several places. Add table type in those places.
• For startup and global invalidate metadata,
  • Approach 1: use getTables API with table type.
    • currently we use IMetaStoreClient#getAllTables(dbName) to get table names inside a db and create IncompleteTables for them. 
    • We can add an additional call to load names of views using IMetaStoreClient#getTables(dbName, tablePattern, tableType).
    • This may increase the startup time by 2x.
  • Approach 2: use getTableMeta API instead of getAllTables.
    • Pros: can get the comments as well. 
    • Cons: This may increase the startup time by 4x.
• Load view's metadata at the end of lightweight DDL operations like "INVALIDATE METADATA table", "CREATE VIEW", etc.
• EventProcessors can extract table type from the event's HMS table object.
• Add a feature flag for this so users don't need this can turn it off to reduce the startup time.

Issue Links

• Dependent: IMPALA-7587 Follow up for IMPALA-2636: consider fixing GetTables() for tables with not loaded metadata
• is related to: IMPALA-9669 loaded views are still returned as tables for GET_TABLES in LocalCatalog mode
• is related to: IMPALA-11392 Document for the pull_table_types_and_comments startup flag
• relates to: IMPALA-9172 Load DB/Table ownership info on start-up.
• links to: Code Review