[HIVE-27499] Add optional table name filter to the HMS notification fetch API

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Labels: pull-request-available
• Assignee: Sai Hemanth Gantasala Assign to me
• Reporter: Quanlong Huang
• Created: 12/Jul/23 12:37
• Updated: 22/Mar/24 12:27
• Resolved: 02/Nov/23 16:47

Description

The current notification fetch API does not support filters on table name:

struct NotificationEventRequest {
    1: required i64 lastEvent,
    2: optional i32 maxEvents,
    3: optional list<string> eventTypeSkipList,
}
NotificationEventResponse get_next_notification(1:NotificationEventRequest rqst)

Engines that cache HMS metadata and file metadata might want to know whether they have synced to the latest events on some given tables.

In order to efficiently fetch events specific to a table, we should enhance the notification fetch API to include an optional table name in the request so that only events for that given table are returned. The NOTIFICATION_LOG table in the HMS should also have an index on table_name so that API calls with the filter do not do a table scan.

With the index added, we can also extend the NotificationEventsCountRequest with an optional table name, and extend the response to carry the largest event id on that table. Currently they are:

struct NotificationEventsCountRequest {
    1: required i64 fromEventId,
    2: required string dbName,
    3: optional string catName,
    4: optional i64 toEventId,
    5: optional i64 limit
}
struct NotificationEventsCountResponse {
    1: required i64 eventsCount,
}
NotificationEventsCountResponse get_notification_events_count(1:NotificationEventsCountRequest rqst)

Similar API for multiple tables (given a table lists) will also be helpful since a query usually involves several tables.

CC Sai Hemanth Gantasala, Naveen Gangam 

Issue Links

• is related to: HIVE-28146 Add positive event type filter to the HMS notification fetch API
• links to: GitHub Pull Request #4569