[IMPALA-8606] GET_TABLES performance in local catalog mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.2.0
• Fix Version/s: Impala 3.3.0
• Component/s: Catalog
• Labels: catalog-v2
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Balazs Jeszenszky
• Created: 31/May/19 13:57
• Updated: 29/Oct/20 17:30
• Resolved: 25/Jul/19 22:06

Description

With local catalog mode enabled, GET_TABLES JDBC requests will return more than the always available table information. Any request for more metadata about a table will trigger a full load of that table on the catalogd side, meaning that GET_TABLES triggers the load of the entire catalog. Also, as far as I can see, the requests for more metadata are made one table at a time.

Once the tables are loaded on the catalogd-side, a coordinator needs 3 roundtrips to the catalog to fetch all the details about a single table. My test case had around 57k tables, 1700 DBs, and ~120k partitions.
GET_TABLES on a cold catalog takes 18 minutes. With a warm catalog, but cold impalad, it still takes ~70 seconds.

Many tools use GET_TABLES to populate dropdowns, etc. so this is bad for both end user experience and catalog memory usage.

Issue Links

• is related to: IMPALA-7587 Follow up for IMPALA-2636: consider fixing GetTables() for tables with not loaded metadata