[IMPALA-11490] More metrics to debug event processing lagging behind

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Component/s: Catalog
• Labels: supportability
• Epic Link: event-processor-supportability
• Target Version: Impala 4.2.0
• Epic Color: ghx-label-10
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 10/Aug/22 02:04
• Updated: 07/Mar/24 19:46
• Resolved: 09/Sep/22 02:35

Description

Event processor could lag behind in many cases, e.g. processing lots events on large tables, waiting for table locks held by manual refresh or other metadata operations, etc.

Currently we have metric on the last synced event id. We should also add metric on the latest event id in HMS. Users can compare them to know whether event processing is lagging behind.

We should also add logs/metrics on tables that take long time in event processing, especially those longer than the event polling interval. So users can decide whether to disable event processing on them, or reduce concurrency of metadata operations on them.

Some metrics like average events processing duration in the last 5min, 30min or 1h will also be helpful.