[IMPALA-6663] Expose current DDL metrics (grouped by type) in the Catalog web UI

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: Impala 2.12.0
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Labels: observability ramp-up supportability
• Epic Link: SupportabilityRoadMap
• Epic Color: ghx-label-5
• Assignee: Tamas Mate
• Reporter: Bharath Vissapragada
• Created: 14/Mar/18 18:54
• Updated: 15/Jul/21 00:07
• Resolved: 09/Mar/20 10:49

Description

It helps if the catalog server can expose currently running (counts) DDL and RESET_METADATA requests via the web UI metrics. From an observability/supportability point of view, these numbers help reason about the Cluster behavior during Catalog heavy load or in situations where Catalog server is the bottleneck

Following are some of the interesting ones that I think are interesting.

enum TDdlType {
  ALTER_TABLE,
  ALTER_VIEW,
  CREATE_DATABASE,
  CREATE_TABLE,
  CREATE_TABLE_AS_SELECT,
  CREATE_TABLE_LIKE,
  CREATE_VIEW,
  CREATE_FUNCTION,
  COMPUTE_STATS,
  DROP_DATABASE,
  DROP_TABLE,
  DROP_VIEW,
  DROP_FUNCTION,
  CREATE_DATA_SOURCE,
  DROP_DATA_SOURCE,
  DROP_STATS,
  CREATE_ROLE,
  DROP_ROLE,
  GRANT_ROLE,
  REVOKE_ROLE,
  GRANT_PRIVILEGE,
  REVOKE_PRIVILEGE,
  TRUNCATE_TABLE,
}

TCatalogOpType.RESET_METADATA
TStmtType.DML

This should be based on the metrics framework added in IMPALA-4886.

Issue Links

• is related to: IMPALA-9118 Add debug page for in-flight DDLs in catalogd
• relates to: IMPALA-7959 Trace and visualize concurrent Catalog operations