[SENTRY-2300] Move Permission Update due to DDL to HMS Post Event Listener

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0, 2.2.0
• Component/s: Sentry
• Assignee: Unassigned
• Reporter: Na Li
• Created: 09/Jul/18 20:23
• Updated: 14/Jun/19 16:40

Description

There was a code in MetastorePlugin that modified Sentry privileges on table Create/Drop and database Create/Drop. As part of Sentry HA work we moved all this logic from Sentry plugin to be driven by notifications which required the extra synchronization between HMS and Sentry.

It should be possible to do permission changes in the post event listener itself to avoid blocking for Sentry. This requires some experiments though because it may cause strange artifacts since at the time these DDL operations are done Sentry may not be aware of the current state - for example you may try to change permissions of a table that Sentry doesn’t know about, which seems to be OK.

This update will have the following benefits:

* HMS waits on Sentry polling HMS update takes 0.5 to 1 second. This update will remove this delay
* Sentry knows every DDL update, and therefore can update permission correctly. In current approach using notification processing, Sentry could miss updates if full snapshot is fetched from HMS, and permission is not updated correctly. In the case of table rename, when mission DDL update event because of full snapshot, sentry will not move the permissions associated with old table to the new table. And the authorization on queries on the renamed table will fail.

The proposed approach is:

1) Add the permission update in SentryPolicyStoreProcessor.sentry_notify_hms_event() through SentrySyncHMSNotificationsPostEventListener for create/drop/alter table, create/drop database commands. This is synchronous processing.
2) Remove the permission update from NotificationProcessor.processNotificationEvent() for create/drop/alter table, create/drop database commands. 
3) Remove the blocking of HMS in SentrySyncHMSNotificationsPostEventListener for Sentry fetching notification events asynchronously from HMS for create/drop/alter table, create/drop database commands. The reason is that now the permission is updated through SentrySyncHMSNotificationsPostEventListener. Therefore, there is no need for HMS to wait for Sentry fetching those notification events. 

Without this approach, the DDL operation take at least 500 to 1000 milliseconds. It is easy for HMS to encounter SyncTimeoutException when it takes long time for Sentry to get full snapshot or fetch notification events. Many users choose to remove the post event listener in configuration (HMS not waiting for Sentry fetching notification events asynchronously) to get rid of this exception. Users cannot take this workaround if they want owner privilege enabled.

This approach makes DDL operation in HMS independent from Sentry fetching full snapshot or notification events, and reduces the DDL operation delay significantly. With this approach, the DDL operation is in the range of few milliseconds. This approach fixes the timeout issue once for all. And users can get rid of timeout issue and enable owner privilege at the same time

Issue Links

• is related to: SENTRY-2396 Minimize the time sentry blocks HMS threads
• links to: Review link for this case