Details
-
Epic
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
Improved self event detection in catalogD
-
ghx-label-10
Description
Problem Statement
Impala catalogd has Events processor which polls metastore events at regular intervals to automatically apply changes to the metadata in the catalogd. However, the current design to detect the self-generated events (DDL/DMLs coming from the same catalogd) have consistency problems which can cause query failures under certain circumstances.
Current Design
The current design of self-event detection is based on adding markers to the HMS objects which are detected when the event is received later to determine if the event is self-generated or not. These markers constitute a serviceID which is unique to the catalogd instance and a catalog version number which is unique for each catalog object. When a DDL is executed, catalogd adds these as object parameters. When the event is received, Events processor checks the serviceID and if the catalog version of the current object with the same name in the catalogd cache and makes a decision of whether to ignore the event or not.
Problems with the current design
The approach is problematic under some circumstances where there are conflicting DDLs repeated at a faster interval . For example, consider the following sequence of alter table operations on a table t1 as per HMS
1. alter table t1 from source s1 say another Impala cluster
2. alter table t1 from source s2 say another Hive cluster
3. alter table t1 from local
The #3 alter table ddl operation would get reflected in the local cache immediately. However, event processor later on would process events from #1 and #2 above and try to alter the table. In an ideal scenario, these alters should have been applied before #3 (i.e in the same order as that in HMS). This leaves table t1 in an inconsistent state.
Proposed Solution
The main idea of the solution is to keep track of the last event id for a given table as eventId which the catalogd has synced to in the Table object. The events processor ignores any event whose EVENT_ID is less than or equal to the eventId stored in the table. Once the events processor successfully processes a given event, it updates the value of eventId in the table before releasing the table lock. Also, any DDL or refresh operation on the catalogd (from both catalog metastore server and Impala shell) will follow the following steps to update the event id for the table:
1. Acquire write lock on the table
2. Perform ddl operation in HMS
3. Sync table till the latest event id (as per HMS) since its last synced event id
The above steps ensure that any concurrent updates applied on a same db/table from multiple sources like Hive, Impala or say multiple Impala clusters, get reflected in the local catalogd cache (in the same order as they appear in HMS) thus removing any inconsistencies.
Also the solution relies on the existing locking mechanism in the catalogd to prevent any other concurrent updates to the table (even via EventsProcessor).
In case of database objects, we will also have a similar eventId which represents the events on the database object (CREATE, DROP, ALTER database) and to which the catalogd as synced to. Since there is no refresh database command, catalogOpExecutor will only update the database eventId when there are DDLs at the database level (e.g CREATE, DROP, ALTER database)
Attachments
Issue Links
- relates to
-
-
- Open
-