Yibing Shi, Hive-7973 comes in multiple parts: 1. HMS schema changes to include notification logs, 2. DbNotificationListener which is essentially a postEventListener, 3. ReplicationTasks which use these notification logs for replication.
For 2 above, In Sentry's plugin for HMS, we have a very similar post event listener implementation (SentryMetastorePostEventListener). The only difference right now being: Sentry's listener sends these notifications as RPC to Sentry service, where as DbNotificationListener persists the notifications in DB.
We think persisting in DB approach is better than RPC as it brings with it multiple advantages mainly 1. Delta changes are persisted allowing the clients (even in a Sentry HA failover case) to pull 2. This gets a global sequence of HMS updates when there are more than one writers (HMS HA). 3. Opportunity to make this part of transaction, so that notification log is a true WAL with consistency guarantees.
Given that, we do have two options here. 1. Either improve Sentry's plugin to persist to Notification Log. 2. Improve Hive plugin
We are inclining more towards option 1, as there are also some Sentry specific optimizations we might be able to do like capture location in the notification log. Which also avoids a second round trip to HMS when processing the notification log, which avoids consistency issues around this.
Also, Sentry's plugin is running on production use cases for a while now, whereas 7973 is pretty new, so it would be a while before it matures. We can always contribute back the Sentry plugin to Hive upstream if there is enough interest. What do you think?