[SENTRY-1669] HMSFollower should read current processed notification ID from database every time it runs

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Hdfs Plugin
• Assignee: Krishna Kalyan
• Reporter: Hao Hao
• Created: 21/Mar/17 00:25
• Updated: 25/Oct/17 18:52
• Resolved: 19/May/17 06:15

Description

Instead of using in-memory current processed notification ID, HMSFollower should read the processed notification ID info from database every time it runs. Otherwise, after failover, the new leader cannot get the correct up-to-date processed notification ID.

It should also update the Notification ID to database once it processes a notification event. The Notification ID in database should keep on increase (we don't handle its overflow scenario).

HMS notification ID's are stored in separate table for couple of reasons

1. SENTRY_PATH_CHANGE is not updated for every notification that is received from HMS. There are cases where HMSFollower doesn't process notifications and skip's them. Depending on SENTRY_PATH_CHANGE information may not provide the last notification processed.
2. There could be cases where HMSFollower thread in multiple sentry servers be acting as a leader and process HMS notifications. we need to avoid processing the notifications multiple times. This can be made sure by always having some number of notification information always regardless of purging interval.
   3. SENTRY_PATH_CHANGE information stored can typically be removed once namenode plug-in has processed the update.

As purpose and usage of notification ID information is different from PATH update info, it locally makes sense to store notification ID separately.

Issue Links

• blocks: SENTRY-1752 HMSFollower gets stuck once it fails to process a notification event
• blocks: SENTRY-1762 notification id's in SENTRY_HMS_NOTIFICATION_ID should be purged periodically
• blocks: SENTRY-1761 Add test for concurrent HMS followers processing notifications
• duplicates: SENTRY-1750 HMSFollower does not handle view update correctly
• is blocked by: SENTRY-1726 sql changes to store last notification-id processed
• is duplicated by: SENTRY-1752 HMSFollower gets stuck once it fails to process a notification event
• is related to: SENTRY-1765 CounterWait.update should be less strict
• is related to: SENTRY-1771 HDFS client concurrently requests full permission update multiple times
• relates to: SENTRY-1649 Initialize HMSFollower when sentry server actually starts
• links to: Code review Link