[HIVE-21637] Synchronized metastore cache

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Target Version/s: 4.0.0
• Assignee: Kishen Das
• Reporter: Daniel Dai
• Created: 21/Apr/19 07:56
• Updated: 18/May/20 04:11

Description

Currently, HMS has a cache implemented by CachedStore. The cache is asynchronized and in HMS HA setting, we can only get eventual consistency. In this Jira, we try to make it synchronized.

Issue Links

• links to: Design Doc
• links to: RB

Sub-Tasks

• [CachedStore] Tear down locks in CachedStore: Open
• [Optimization ] Do not open transaction for readonly query: Open
• [ Test-fix ] alter_table_update_status/alter_table_update_status_disable_bitvector/alter_partition_update_status fail when DbNotificationListener is installed: Open
• [RawStore] RawStore changes to facilitate HMS cache consistency: Open
• [CachedStore ] Prewarm HMS cache during bootstrap with ValidWriteIdList for all the tables: In Progress
• [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper: In Progress
• [HMS] Make all Metastore reads consistent: Open
• [CachedStore] Use notification log to keep the cache up-to-date with all the changes: Open
• [HMS Client ] During writes, mark writeId as committed locally in HMS client: Open
• [CachedStore] Implement caching/fetching of foreign keys based on parent db/table: Open
• [cachedstore] Add tableId and writeId check in cachedStore: Open
• Future ALTER TABLE flows should advance write Id for transactional table: In Progress