[IMPALA-12399] Pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest to avoid receiving OPEN_TXN events from HMS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-4
• Assignee: Venugopal Reddy K
• Reporter: Venugopal Reddy K
• Created: 24/Aug/23 07:25
• Updated: 27/Mar/24 19:00
• Resolved: 27/Mar/24 19:00

Description

Notification events like OPEN_TXN are ignored on catalogd MetastoreEventsProcessor. So, we can pass eventTypeSkipList with OPEN_TXN in NotificationEventRequest while invoking get_next_notification() to avoid reading such notification messages from HMS and then ignoring on catalogd. OPEN_TXN event being more frequent(received even upon describe table operation from beeline), we can significantly reduce unwanted processing on both HMS and catalogd. Catalogd reads events in batches of EVENTS_BATCH_SIZE_PER_RPC, skipping such unnecessary events can help catchup the events faster.

Issue Links

• is related to: IMPALA-12933 Catalogd should set eventTypeSkipList when fetching specifit events for a table