[IMPALA-10794] Improve observability of execDdl code paths

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0, Impala 4.1.1
• Component/s: Catalog, Frontend
• Labels: observability supportability
• Epic Color: ghx-label-7
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 15/Jul/21 00:13
• Updated: 08/Jun/23 16:35
• Resolved: 08/Aug/22 23:51

Description

It's hard to debug a hanging DDL/DML:

• The last log before coordinator sending execDdl RPC to catalogd is "Analysis and authorization finished."
• Catalogd just prints logs when it finishes the HMS operations and start reloading metadata incrementally.

It'd be helpful to add logs:

• On coordinator side, add logs when start/finish the execDdl RPCs.
• On catalogd side, add logs when start/finish executing an execDdl request. More loggings in specifit complex ddls will also be helpful.

We also need to add thread annotation in catalogd for the part executing HMS RPCs. We currently only have them in loading metadata. That's not enough.

CC vihangk1 who may already have a draft for this.

Issue Links

• relates to: IMPALA-7450 catalogd should use thread names to make jstack more readable