[IMPALA-10926] Sync db/table to latest event id for ddl operations in catalog metastore server

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Epic Link: Improved self event detection in catalogD
• Epic Color: ghx-label-7
• Assignee: Sourabh Goyal
• Reporter: Sourabh Goyal
• Created: 20/Sep/21 22:50
• Updated: 23/Feb/22 20:24
• Resolved: 23/Feb/22 20:24

Description

Whenever DDL operations like add/remove partitions, alter table etc are performed from catalog HMS endpoints, we will do the following: 

1. Acquire lock on db/table 
2. Perform HMS operation
3. For db/table in step 1, fetch all events from HMS
4. Apply those events on the db/table in catalogd cache 
5. Set the last processed event id for db/table 

In addition to the above, the following needs to be handled as well: 

1. Event processor should skip processing an event on a db/table if the db/table is already synced till that event id. 
2. If last synced event id in table/db is less than the event being processed by event processor, it should set the last synced db/table event id after successfully processing the event. 
3. Full table refresh should set the last event processed. 

cc - kishendas vihangk1

Issue Links

• Dependent: IMPALA-10949 Improve batching logic of events