[IMPALA-9118] Add debug page for in-flight DDLs in catalogd

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Labels: observability supportability
• Epic Link: Catalog Observability
• Epic Color: ghx-label-9
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 04/Nov/19 08:04
• Updated: 06/Mar/24 13:26
• Resolved: 26/Oct/23 00:30

Description

In a busy cluster, it's possible that many DDL/DML queries keep in the CREATED state for several minutes. Especially when using with sync_ddl=true, tens of minutes are also possible. They may be waiting for the ExecDdl RPC to catalogd to finish.

It'd be helpful for debugging DDL/DML hangs if we can show the in-flight DDLs in catalogd. I think the following fields are important:

• thread id
• coordinator
• db name / table name
• ddl type, e.g. AddPartition, DropTable, CreateTable, etc. More types here.
• last event, e.g. waiting for table lock, got table lock, loading file metadata, waiting for sync ddl version etc.
• start time
• time elapsed
• (optional) params link to show the TDdlExecRequest in json format

It'd be better to also include running REFRESH/INVALIDATE METADATA commands 

Issue Links

• causes: IMPALA-12687 Key conflicts in catalog inFlightOperations when catalog RPC retries
• is related to: IMPALA-12778 Catalog operations page is showing UTC time strings
• is related to: IMPALA-6557 Show details of recent topic delta update
• relates to: IMPALA-7959 Trace and visualize concurrent Catalog operations
• relates to: IMPALA-6663 Expose current DDL metrics (grouped by type) in the Catalog web UI