[IMPALA-11409] Skip UpdateCatalogMetrics if another thead is on-going in it

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Won't Do
• Component/s: Backend
• Epic Color: ghx-label-10
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 01/Jul/22 12:40
• Updated: 26/Mar/24 07:16
• Resolved: 26/Sep/23 05:38

Description

Impala coordinator tracks local metrics of the catalog, e.g. number of dbs/tables. When use_local_catalog is enabled, it also tracks the cache metrics, e.g. cache hit/miss count/rate.

These metrics are updated at the end of each statement, even for simple statements like "USE <db>", "SET var=xxx", "SELECT 1". The catalog update thread will also update the metrics in the end.
https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L1272
https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L2065

These metrics are global metrics of the local catalog cache. They are not specifit to a single statement. It's a waste to update the metrics concurrently.
https://github.com/apache/impala/blob/bb610dee09a8069bb993b4c668f7e481c1774b70/be/src/service/impala-server.cc#L1526-L1559

We've seen "hanging issues" that all statements, including the catalog update thread, are slowly executing the UpdateCatalogMetrics() function. See details in the attached jstack dump.

Indeed, if one thread is running the UpdateCatalogMetrics() function, the other threads can skip it and move forward.

Issue Links

• is related to: IMPALA-12699 Coordinator should retry GetPartialCatalogObject request and apply a recv timeout
• relates to: IMPALA-8675 Remove db/table count metrics from impalad in LocalCatalog mode