[HIVE-21637] Synchronized metastore cache

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Target Version/s: 4.0.0
• Assignee: Kishen Das
• Reporter: Daniel Dai
• Created: 21/Apr/19 07:56
• Updated: 18/May/20 04:11

Description

Currently, HMS has a cache implemented by CachedStore. The cache is asynchronized and in HMS HA setting, we can only get eventual consistency. In this Jira, we try to make it synchronized.

Issue Links

• links to: Design Doc
• links to: RB

Sub-Tasks

• [CachedStore] Tear down locks in CachedStore: Open
• [CachedStore] Add table constraints in CachedStore: Closed
• [Optimization ] Do not open transaction for readonly query: Open
• [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637: Closed
• [ Interface changes ] Add table id to HMS get methods: Resolved
• [ Test-fix ] alter_table_update_status/alter_table_update_status_disable_bitvector/alter_partition_update_status fail when DbNotificationListener is installed: Open
• [RawStore] RawStore changes to facilitate HMS cache consistency: Open
• [CachedStore ] Prewarm HMS cache during bootstrap with ValidWriteIdList for all the tables: In Progress
• [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper: In Progress
• [HMS] Make all Metastore reads consistent: Open
• [HMS] Advance the write id for the table for DDL: Closed
• [CachedStore] Use notification log to keep the cache up-to-date with all the changes: Open
• [HMS Client ] During writes, mark writeId as committed locally in HMS client: Open
• [CachedStore] Add check/default constraints in CachedStore: Closed
• Send ValidWriteIDList in request for all the new HMS get_* APIs that are in request/response form: Closed
• [CachedStore] Implement caching/fetching of foreign keys based on parent db/table: Open
• [HS2] Send tableId in request for get_table_request API: Closed
• Send tableId in request for all the new HMS get_partition APIs: Resolved
• [CachedStore] Add flag in TableWrapper in CacheStore to check if constraints are set or not: Closed
• Send ValidWriteIdList and tableId to get_*_constraints HMS APIs: Closed
• [CachedStore] Data mismatch between CachedStore and ObjectStore for constraints: Closed
• [CachedStore] Constraints read from cache should be consistent snapshot.: Closed
• [cache store] Add valid flag in table wrapper for all constraint: Resolved
• Advance write Id during AlterTableAddConstraint DDL: Resolved
• Advance write Id during AlterTableDropConstraint DDL: Resolved
• [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2: Resolved
• [HMS] Populate tableId in the response of get_valid_write_ids API: Resolved
• [HS2] Cache tableId in SessionState: Resolved
• [HS2] Enhance DriverTxnHandler.isValidTxnListState logic to include tableId comparison: Resolved
• [HMS] Provide new HMS API to return latest committed compaction record for a given table: Closed
• Provide default implementation for HMS APIs: Closed
• [cachedstore] Add tableId and writeId check in cachedStore: Open
• [HMS] Advance write Id during AlterTableDropPartition: Closed
• [Hive] Advance write ID for remaining DDLs: Resolved
• Advance Write ID during ALTER TABLE ( NOT SKEWED, SKEWED BY, SET SKEWED LOCATION, UNSET SERDEPROPERTIES): Closed
• Future ALTER TABLE flows should advance write Id for transactional table: In Progress
• Add a test case to ensure Truncate table advances the write ID: Closed
• ValidWriteIdList & table id are sometimes missing when requesting partitions by name via HS2: Closed
• Add unit tests for Hive#getPartitionsByNames using batching: Closed
• Drop unused requests from TestHiveMetaStoreClientApiArgumentsChecker: Closed