[IMPALA-12699] Coordinator should retry GetPartialCatalogObject request and apply a recv timeout

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Epic Color: ghx-label-5
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 10/Jan/24 06:33
• Updated: 26/Mar/24 07:24
• Resolved: 26/Mar/24 07:24

Description

We have seen trivial GetPartialCatalogObject RPCs hanging in coordinator side, e.g. IMPALA-11409. Due to the piggyback mechanism of fetching metadata in local-catalog mode (see IMPALA-7534 or comments in CatalogdMetaProvider#loadWithCaching()), a hanging RPC on shared metadata (e.g. db list or table list of a db) could block other queries.

We have also seen thrift RPCs hanging in IMPALA-3575. In fact, GetPartialCatalogObject RPCs are read-only requests. They can be cleanly retried. We should consider using a dedicated catalogd client cache for GetPartialCatalogObject requests and set an appropriate timeout for the socket.

The current catalogd client cache:
https://github.com/apache/impala/blob/cdac777c51febc99500b8426c2b3aabc7e9addd7/be/src/runtime/exec-env.cc#L224-L226
The related flags:
https://github.com/apache/impala/blob/cdac777c51febc99500b8426c2b3aabc7e9addd7/be/src/runtime/exec-env.cc#L161-L167

CC wzhou

Issue Links

• relates to: IMPALA-7534 Handle invalidation races in CatalogdMetaProvider cache
• relates to: IMPALA-3575 Impala should retry backend connection request and apply a send timeout
• relates to: IMPALA-11409 Skip UpdateCatalogMetrics if another thead is on-going in it