[SENTRY-872] Uber jira for HMS HA + Sentry HA redesign

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.0, 2.0.0
• Fix Version/s: 2.0.0
• Component/s: Hdfs Plugin
• Assignee: Alex Kolbasov
• Reporter: Sravya Tirukkovalur
• Created: 02/Sep/15 22:57
• Updated: 27/Nov/17 13:32
• Resolved: 21/Nov/17 14:38

Issue Links

• contains: SENTRY-871 Refactor HA components based on Sentry-852
• contains: SENTRY-873 [HMS HA] Have a HMS leader which would be responsible for sending path updates to Sentry
• is blocked by: SENTRY-1596 Hive tests failing for sentry-ha-redesign branch
• is blocked by: SENTRY-1759 UpdatableCache leaks connections
• is blocked by: SENTRY-1323 Bump the hive version to 1.2.0
• is blocked by: SENTRY-1474 createSentryRole() isn't thread-safe
• is blocked by: SENTRY-1476 SentryStore is subject to JDQL injection
• is blocked by: SENTRY-1486 Sentry should use repeatable-read consistency level
• is blocked by: SENTRY-1491 Sentry transactions are not rolled back immediately when commit fails
• is blocked by: SENTRY-1644 Partition ACLs disappear after renaming Hive table with partitions
• is blocked by: SENTRY-1681 SentryHdfsServiceException is an unchecked exception
• is blocked by: SENTRY-1120 Show role / privileges info in Sentry Service Webpage
• is blocked by: SENTRY-1507 Sentry should use Datanucleus version of javax.jdo
• is related to: SENTRY-852 Create PathUpdateForwarder and PermUpdateForwarder
• is related to: SENTRY-1314 HDFS Sync improvements
• is related to: SENTRY-1589 Flaky test TestDbPrivilegeCleanupOnDrop.testReCreateObjects
• is related to: SENTRY-1508 MetastorePlugin.java does not handle properly initialization failure
• is related to: SENTRY-1500 Sentry should use Datanucleus version of javax.jdo
• is related to: SENTRY-1506 Importing JDO or datanucleus properties from config is broken
• is related to: SENTRY-1523 sentry-dist is missing dependency on mysql connector
• is related to: SENTRY-1526 Sentry processed stays alive after being killed
• is related to: SENTRY-1644 Partition ACLs disappear after renaming Hive table with partitions
• is related to: SENTRY-1699 MetastoreCacheInitializer shouldn't use Preconditions for HMS data
• is related to: SENTRY-1788 Sentry Store may use JDO object after the associated data is removed in DB
• is related to: SENTRY-1797 SentryKerberosContext should use periodic executor instead of managing periodic execution via run() method.
• is related to: SENTRY-1801 Sentry Namenode Plugin should handle unknown permissions
• is related to: SENTRY-1915 Sentry is doing a lot of work to convert list of paths to HMSPaths structure
• is related to: SENTRY-1916 Sentry should not store paths outside of the prefix
• is related to: SENTRY-1574 SentryMetastorePostEventListener class is not used by anything
• is related to: SENTRY-1524 sentry-dist is missing dependency on sentry-hdfs-dist
• is related to: SENTRY-1597 Document Sentry protocols
• is related to: SENTRY-1653 Dump Sentry metrics to a file periodically
• is related to: SENTRY-1494 Evaluate using a JSON notification log message as a Map rather than using a Deserializer with Java message object.
• is related to: SENTRY-1537 Investigate changing ROLE:PRIV model from M:N to 1:N
• is related to: SENTRY-1579 Sentry should avoid using Proxy
• is related to: SENTRY-627 Extend Sentry HA to work with Hive Metastore HA and HDFS
• is related to: SENTRY-1501 SentryStore shouldn't synchronize openTransaction() and commitUpdateTransaction()
• is related to: SENTRY-1518 Add metrics for SentryStore transactions
• is related to: SENTRY-1525 Provide script to run Sentry directly from the repo
• is related to: SENTRY-1573 SentryMetastoreListenerPlugin should be an interface
• is related to: SENTRY-1820 Add JSON file reporter for Sentry metrics
• is related to: SENTRY-1867 DataNucleus.Query INFO level logging spams Sentry log files
• is related to: SENTRY-1531 Add transaction metrics for SentryStore
• is related to: SENTRY-1594 TransactionBlock should become generic
• is related to: SENTRY-1540 SentryStore.isMultiActionsSupported() is always true
• is related to: SENTRY-1503 Remove all sequence ID checks from TestSentryStore
• is related to: SENTRY-1502 Multiple tests in TestSentryStore are not testing useful things
• is related to: SENTRY-1482 Evaluate 4000 char limit for URI and semantics around overflow
• Parent Feature: SENTRY-1721 Unit test failures in TestSentryStore due to changeId miscount
• Parent Feature: SENTRY-1771 HDFS client concurrently requests full permission update multiple times
• relates to: SENTRY-1440 Fencing implementation in sentry-ha can create two fencing tables
• relates to: SENTRY-880 Delete the path cache children as soon as it is processed.
• relates to: SENTRY-883 [HA] Pre existing children in the path cache trigger child added events
• relates to: SENTRY-876 Handle path updates when Sentry HA + HDFS sync - Send updates to all sentry instances
• relates to: SENTRY-1481 Evaluate use of paths in MAuthzPathsMapping
• relates to: SENTRY-662 SentryServiceIntegrationBase should use UGI based login
• relates to: SENTRY-870 Create UpdateForwarders for paths and permissions
• relates to: SENTRY-882 HMS leader should handle out of order updates when HA
• relates to: SENTRY-1315 Add an interface in WebUI to request for a Sentry full update
• relates to: SENTRY-1322 Persist HMSPaths full update into Sentry DB
• relates to: SENTRY-877 Recreate the metastore cache if ZK connection is lost.
• links to: Review link

Sub-Tasks

• Add metrics for isActive and isHA: Resolved
• Implement Sentry leadership election: Resolved
• Store HMSPaths in Sentry DB to allow fast failover: Resolved
• Upgrading SQL script for HMSPaths persistence: Resolved
• Implement fencing required for active/standby: Resolved
• Add sentry specific test cases to use NotificationLog: Resolved
• Adapt SentryMetaStorePostEventListener to write HMS notification logs: Resolved
• Fix TestLeaderStatus#testRacingClients: Resolved
• Move SentryHDFSServiceClient code from hdfs-common into hdfs-service: Resolved
• Integrate Fencer with SentryStore: Resolved
• The sentry client should retry RPCs if it gets a SentryStandbyException (SentryPolicyServiceClient - pool version): Resolved
• Move the HDFS code which lives inside the sentry daemon into sentry-provider: Resolved
• Implement HMSFollower in Sentry service which reads the NotificationLog entries: Resolved
• Handle updating HMSState for HDFS plugin in HMSFollower: Resolved
• Rework fetching Hive Paths state: Resolved
• Add metrics for Sentry HA: Resolved
• Make HiveConf and Hive client jars available to Sentry deamon: Resolved
• Notify Sentry about HMS new notifications if low delay is desired: Resolved
• Changes to get the Fencer working with Oracle and MySQL: Resolved
• Evict datanucleus second-level cache during activation: Resolved
• [Test infra] Ability to start active and standby sentry service using InternalSentrySrv: Resolved
• [Test hook] Provide a hook to stop the active sentry sevice: Resolved
• Support intializing state in Sentry service: Resolved
• JDO deadlocks while processing grant while a background thread processes Notificationlogs: Resolved
• May want to disallow reads on Sentry passive: Resolved
• Do not start up HMSFollower if hive is not using Sentry: Resolved
• Test TGT renewals in HMSFollower: Resolved
• Only leader should follow HMS updates: Resolved
• Test Sentry HA Tasks: Resolved
• GenericServiceClient should support connection pools: Resolved
• Sentry should not serve requests until the full update processing is finished: Resolved
• Error during fencing table rename can disable master: Resolved
• Fencing implementation in sentry-ha can create two fencing tables: Resolved
• Store processed notification sequence ID in database: Resolved
• Rebase sentry-ha-redesign branch on master: Resolved
• Ensure HMS point-in-time snapshot consistency: Resolved
• Sentry clients should retry with another server when they get connection errors: Resolved
• Disable fencing in Sentry store for Active/Active: Resolved
• HMS plugin should wait until Sentry handles the update before continuing.: Resolved
• Renaming SQL script for HMSPaths persistence: Resolved
• Add feature flag for using NotifcationLog: Resolved
• Add option to use non pool model for sentry client: Resolved
• HDFS Sync change for handling persisted Sentry delta or full updates: Resolved
• HMS Follower thread should terminate when Sentry receives ^C: Resolved
• HMS Follower should update HDFS plugin paths: Resolved
• Refactor SentryStore transaction management to allow for extra TransanctionBlocks for a single permission update: Resolved
• Create schema for storing HMS path change and Sentry permission change.: Resolved
• HMS Follower should store arriving HMS notifications: Resolved
• Remove fencing support: Resolved
• Add feature flag to allow stand-alone configuration without ZK: Resolved
• Make full Perm/Path snapshot available for NN plugin: Resolved
• Refactor propagating logic for Perm/Path delta to NN plugin: Resolved
• Upgrading SQL scripts for persist Perm/Path change: Resolved
• Suport secure ZK configuration for leader election: Resolved
• Provide pooled client connection model with HA: Resolved
• Refactor ZK/Curator code: Resolved
• Refactor SentryStore transaction to persist a single path transcation bundled with corresponding delta path change: Resolved
• Backport SENTRY-1404 to Sentry-ha-redesign branch: Resolved
• Implement NN client failover for Sentry HA: Resolved
• Implement client failover for Generic and NN clients: Resolved
• Hive tests failing for sentry-ha-redesign branch: Resolved
• Define Thrift API for HMS to Sentry notification barrier: Resolved
• Implement HMS Notification barrier on the server side: Resolved
• Converting Sentry to a stateless service: Resolved
• Purge MSentryPerm/PathChange tables: Resolved
• HMSFollower should persist full HMS snapshot into SentryDB if there is not one.: Resolved
• Add propagating logic for Perm/Path updates in Sentry service: Resolved
• Fetch Hive Paths point-in-time full snapshot during Sentry startup: Resolved
• Fix the secure HMS connection code in HMSFollower: Resolved
• Incorrect usage of AuthzConfVars.AUTHZ_SERVER_NAME may cause HS2 HA not work: Resolved
• HMSFollower to retry connecting to HMS upon connection loss: Resolved
• Typo for notification log feature flag: Resolved
• In HMSFollower failing of catching error causes the executor to halt: Resolved
• Current MAuthzPathsMapping table definition may cause error 'Duplicate entry XX for key PRIMARY': Resolved
• out of sequence error in HMSFollower: Resolved
• Make HMSFollower initialDelay and run period configurable: Resolved
• HMSFollower should not check isLoadMetastoreConfig when trying to connect to HMS: Resolved
• Limit HMS connections only to the leader of the sentry servers: Resolved
• Update SQL script of MSentryPathChange table to add a column for notification ID: Resolved
• Periodically purge Delta change tables.: Resolved
• Cleanup creation of SentryStore and HMSFollower: Resolved
• AutoIncrement ChangeID of MSentryPermChange/MSentryPathChange may be error-prone: Resolved
• Initialize HMSFollower when sentry server actually starts: Resolved
• Port SENTRY-1360 to sentry-ha-redesign: Resolved
• TestHDFSIntegrationAdvanced timeouts on sentry-ha-redesign branch: Resolved
• HMSFollower should read current processed notification ID from database every time it runs: Resolved
• Expose current HMS notification ID as a Sentry gauge (metric): Resolved
• Provide HMSFollower healthcheck (metric): Resolved
• Expose HMS data via Sentry web UI: Resolved
• Improve error reporting from FullUpdateInitializer: Resolved
• HMSFollower shouldn't print the same value of notification ID multiple times: Resolved
• FullUpdateInitializer#createInitialUpdate should not throw RuntimeException: Resolved
• sentry-hdfs-dist should include sentry-core-common after refactor SentryHDFSServiceClientDefaultImpl: Resolved
• Add metrics to measure how much time to get Delta Path/Perm Updates: Resolved
• MetastoreCacheInitializer is lo longer used and should be removed: Resolved
• Investigate use of EXPORT for replication for initial HMS snapshot: Resolved
• FullUpdateInitializer has a race condition in handling results list: Resolved
• Port SENTRY-1489 to sentry-ha-redesign branch: Resolved
• Port SENTRY-1548 to sentry-ha-redesign branch: Resolved
• FullUpdateInitializer can be more efficient: Resolved
• Refactor HA components based on Sentry-852: Resolved
• sql changed needed for AUTHZ_PATH table: Resolved
• Move thrift waiters gauge away from SentryStore: Resolved
• HMSFollower should handle adding a view with empty path.: Resolved
• Deprecate feature flag for enabling notification log: Resolved
• Waiting for HMS notifications from Thrift should be interruptible: Resolved
• Expose time spent creating the initial snapshot as a metric: Resolved
• FullUpdateInitializer should not use preconditions to verify HMS data: Resolved
• Do not start HMSFollower if Hive isn't configured: Resolved
• Avoid randomizing the servers at client side based on configuration.: Resolved
• HMSFollower shouldn't call processNotificationEvents() unless there are events: Resolved
• Enable TestHDFSIntegrationEnd2End.testEnd2End: Resolved
• Disable HMSFollower when HMS integration is not enabled: Resolved
• HMSFollower doesn't need to save path info when HDFS sync is disabled: Resolved
• Sentry should emit log messages when it is ready to serve requests.: Resolved
• TestSentryStore often fails in setup(): Resolved
• Implement alternative HMS/Sentry synchronization: Resolved
• Improve retry handling for FullUpdateInitializer: Resolved
• Sentry HDFS Sync should survive in presence of bad paths objects: Resolved
• Sentry HA Test: programmatic failover in a mini cluster env; also add some test data.: Resolved
• Unit test failures in TestSentryStore due to changeId miscount: Resolved
• Create HMSFollower when SentryService.Start() is called: Resolved
• HDFS e2e tests should wait for HMSFollower to start: Resolved
• Remove old PoolClientInvocationHandler: Resolved
• Include response status in TSentrySyncIDResponse: Resolved
• PathsUpdate.parsePath() calls FileSystem.getDefaultUri() way too often: Resolved
• Test secure ZK connections: Resolved
• Test concurrent roles/groups/privs operations: Resolved
• Create/Alter/Drop database/table should check corresponding property before drop privileges: Resolved
• Generic service client should support Kerberos: Resolved
• SentryTransportFactory may use incorrect kerberos principal: Resolved
• Inefficient connection management by retrying invocation handler: Resolved
• HMSFollower doesn't handle INSERT operation: Resolved
• HMSFollower shouldn't create local hive during tests: Resolved
• HMSFollower gets stuck once it fails to process a notification event: Resolved
• Add HMSFollower per-operation metrics: Resolved
• sql changes to store last notification-id processed: Resolved
• Passive nodes should still follow latest notification ID: Resolved
• Avoid using local hive meta store with wrong configuration: Resolved
• Improve Sentry memory usage by interning object names: Resolved
• add sentry ha e2e test back accommodating to the re-design: Resolved
• HMSFollower should detect when a full snapshot from HMS is required: Resolved
• Add log message for key store file path: Resolved
• notification id's in SENTRY_HMS_NOTIFICATION_ID should be purged periodically: Resolved
• HMSFollower should check for leader status after each event processed: Resolved
• Fix the config string for server load balancing: Resolved
• CounterWait.update should be less strict: Resolved
• HMSFollower should not persist empty full snapshot: Resolved
• Generic model clients using kerberos can no longer connect to Sentry server: Resolved
• Multiple followers should not create full snapshot: Resolved
• Avoid detaching object on transaction exit when it isn't needed: Resolved
• Refactor HMSFollower Class: Resolved
• Avoid more detaches on commit: Resolved
• HDFS client concurrently requests full permission update multiple times: Resolved
• Permissions created before table creation are not reflected in HDFS ACLs: Resolved
• Delta change cleaner should leave way more then a single entry intact: Resolved
• HMSFollower should always depend on persisted information to decide is full snapshot is needed: Resolved
• SentryStore should clear SENTRY_HMS_NOTIFICATION_ID while clearing store: Resolved
• Generic clients are not able to connect to sentry server with kerberos enabled.: Resolved
• FullUpdateInitializer does not kill the threads whenever getFullHMSSnapshot throws an exception: Resolved
• Persist new HMS snapshots with a new generation ID.: Resolved
• Add an HMS image ID to the thrift schema definition for hdfs/sentry requests: Resolved
• DBUpdateForwarder returns empty update list to HDFS instead of full update: Resolved
• Sentry Clients should not log every connection request: Resolved
• Sentry Clients failover not working with kerberos enabled: Resolved
• HMSFollower does not handle view update correctly: Resolved
• Ensure DB to sort delta changes by CHANGE_ID: Resolved
• HMSFollower not persisting last processed notifications when partition is altered: Resolved
• Reenable ignored unit tests from TestHDFSIntegrationEnd2End: Resolved
• Delta tables should not have holes: Resolved
• Add better debug logging for retrieving the delta changes: Resolved
• Provide names for HMSFollower and cleaner threads: Resolved
• Fix flaky HDFS END2END tests: Resolved
• Flaky testConcurrentUpdateChanges test: Resolved
• HMSFollower should handle the case of multiple notifications with the same ID: Resolved
• Sentry server can be more efficient in handling full snapshot from HMS: Resolved
• Define a DB schema for HMS generation IDs: Resolved
• Improve memory handling for HDFS sync: Resolved
• NotificationProcessor may put the wrong path in the update: Resolved
• Provide unit test for LeaderStatusMonitor: Resolved
• Send new HMS snapshots to HDFS requesting an old generation ID: Resolved
• Deprecate SENTRY_HA_ENABLED and all tests that use it: Resolved
• HMSFollower should be a singleton: Resolved
• Transactions could fail to commit to the database under load: Resolved
• SentryStore may serialize transactions that rely on unique key: Resolved
• Dropping a Hive database/table doesn't cleanup the permissions associated with it: Resolved
• Rename version in sentry-ha-redesign branch to 2.0.0-SNAPSHOT: Resolved
• Sentry e2e tests should enable SentrySyncHMSNotificationsPostEventListener: Resolved
• Expose current set of IDs as Sentry metrics: Resolved
• Fix build failures when hive-authz2 profile is enabled.: Resolved
• Separate legacy sentry configs from sentry ha configs for api compatibility: Resolved
• HMSFollower should handle notifications even if HDFS sync is disabled.: Closed
• Revert HMSFollower refactoring change: Resolved
• Persisting HMS snapshot and the notification-id to database in same transaction: Resolved
• Try to use pool with idle connections first: Resolved
• Sentry e2e tests are trying to test without notification log: Resolved
• Sentry should handle the case of multiple notifications with the same ID: Resolved