[IMPALA-13293] LocalCatalog's waitForCatalogUpdate() sleeps too much

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0
• Component/s: Catalog, Frontend
• Epic Color: ghx-label-12
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 09/Aug/24 14:05
• Updated: 16/Aug/24 11:11
• Resolved: 16/Aug/24 11:11

Description

Unlike ImpaladCatalog's waitForCatalogUpdate() the LocalCatalog version doesn't use a conditional variable and simply waits for timeoutMs. The timeout comes from MAX_CATALOG_UPDATE_WAIT_TIME_MS, which is 2 seconds. This means the the function will wait 2 seconds even it the catalog update arrived in the meantime. This 2 seconds is often nearly completely added to the impala cluster startup time.

The sleep was added in https://gerrit.cloudera.org/#/c/11472/3/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java

Update: realized that this also doesn't work well for ImpaladCatalog - the FeCatalogManager creates a new ImpaladCatalog when a full topic update arrives, so the Object that waitForCatalogUpdate() waits for is never notified. My impression is that was broken a long time ago, even before LocalCatalog was added.

Issue Links

• relates to: IMPALA-11729 Investigate and improve impalad startup time