Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21637

Synchronized metastore cache

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Currently, HMS has a cache implemented by CachedStore. The cache is asynchronized and in HMS HA setting, we can only get eventual consistency. In this Jira, we try to make it synchronized.

      Attachments

        1. HIVE-21637.10.patch
          1.86 MB
          Daniel Dai
        2. HIVE-21637.11.patch
          1.86 MB
          Daniel Dai
        3. HIVE-21637.12.patch
          1.86 MB
          Daniel Dai
        4. HIVE-21637.13.patch
          1.90 MB
          Daniel Dai
        5. HIVE-21637.14.patch
          1.90 MB
          Daniel Dai
        6. HIVE-21637.15.patch
          1.91 MB
          Daniel Dai
        7. HIVE-21637.16.patch
          1.91 MB
          Daniel Dai
        8. HIVE-21637.17.patch
          1.91 MB
          Daniel Dai
        9. HIVE-21637.18.patch
          1.91 MB
          Daniel Dai
        10. HIVE-21637.19.patch
          1.91 MB
          Daniel Dai
        11. HIVE-21637.19.patch
          1.0 kB
          Daniel Dai
        12. HIVE-21637.2.patch
          1.04 MB
          Daniel Dai
        13. HIVE-21637.20.patch
          1.92 MB
          Daniel Dai
        14. HIVE-21637.21.patch
          1.92 MB
          Daniel Dai
        15. HIVE-21637.22.patch
          1.92 MB
          Daniel Dai
        16. HIVE-21637.23.patch
          1.93 MB
          Daniel Dai
        17. HIVE-21637.24.patch
          1.95 MB
          Daniel Dai
        18. HIVE-21637.25.patch
          1.95 MB
          Daniel Dai
        19. HIVE-21637.26.patch
          1.95 MB
          Daniel Dai
        20. HIVE-21637.27.patch
          1.99 MB
          Daniel Dai
        21. HIVE-21637.28.patch
          2.00 MB
          Daniel Dai
        22. HIVE-21637.29.patch
          2.00 MB
          Daniel Dai
        23. HIVE-21637.3.patch
          1.04 MB
          Daniel Dai
        24. HIVE-21637.30.patch
          2.00 MB
          Daniel Dai
        25. HIVE-21637.31.patch
          2.00 MB
          Daniel Dai
        26. HIVE-21637.32.patch
          2.01 MB
          Daniel Dai
        27. HIVE-21637.33.patch
          2.01 MB
          Daniel Dai
        28. HIVE-21637.34.patch
          2.01 MB
          Daniel Dai
        29. HIVE-21637.35.patch
          2.01 MB
          Daniel Dai
        30. HIVE-21637.36.patch
          2.04 MB
          Daniel Dai
        31. HIVE-21637.37.patch
          2.12 MB
          Daniel Dai
        32. HIVE-21637.38.patch
          2.12 MB
          Daniel Dai
        33. HIVE-21637.39.patch
          2.16 MB
          Daniel Dai
        34. HIVE-21637.4.patch
          1.08 MB
          Daniel Dai
        35. HIVE-21637.40.patch
          2.16 MB
          Daniel Dai
        36. HIVE-21637.41.patch
          2.16 MB
          Daniel Dai
        37. HIVE-21637.42.patch
          2.15 MB
          Daniel Dai
        38. HIVE-21637.43.patch
          2.16 MB
          Daniel Dai
        39. HIVE-21637.44.patch
          2.16 MB
          Daniel Dai
        40. HIVE-21637.45.patch
          2.12 MB
          Daniel Dai
        41. HIVE-21637.46.patch
          2.16 MB
          Daniel Dai
        42. HIVE-21637.47.patch
          2.16 MB
          Daniel Dai
        43. HIVE-21637.48.patch
          2.16 MB
          Daniel Dai
        44. HIVE-21637.49.patch
          2.16 MB
          Daniel Dai
        45. HIVE-21637.5.patch
          1.08 MB
          Daniel Dai
        46. HIVE-21637.50.patch
          2.16 MB
          Daniel Dai
        47. HIVE-21637.51.patch
          2.14 MB
          Daniel Dai
        48. HIVE-21637.52.patch
          2.16 MB
          Daniel Dai
        49. HIVE-21637.53.patch
          2.16 MB
          Daniel Dai
        50. HIVE-21637.54.patch
          2.16 MB
          Daniel Dai
        51. HIVE-21637.55.patch
          2.14 MB
          Daniel Dai
        52. HIVE-21637.56.patch
          2.14 MB
          Daniel Dai
        53. HIVE-21637.57.patch
          2.14 MB
          Daniel Dai
        54. HIVE-21637.58.patch
          2.14 MB
          Daniel Dai
        55. HIVE-21637.59.patch
          2.14 MB
          Daniel Dai
        56. HIVE-21637.6.patch
          1 kB
          Daniel Dai
        57. HIVE-21637.60.patch
          2.14 MB
          Daniel Dai
        58. HIVE-21637.61.patch
          2.14 MB
          Daniel Dai
        59. HIVE-21637.7.patch
          1.78 MB
          Daniel Dai
        60. HIVE-21637.8.patch
          1.80 MB
          Daniel Dai
        61. HIVE-21637.9.patch
          1.86 MB
          Daniel Dai
        62. HIVE-21637-1.patch
          247 kB
          Daniel Dai

        Issue Links

          1.
          [CachedStore] Tear down locks in CachedStore Sub-task Open Unassigned  
          2.
          [CachedStore] Add table constraints in CachedStore Sub-task Closed Adesh Kumar Rao

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 40m
          3.
          [Optimization ] Do not open transaction for readonly query Sub-task Open Unassigned  
          4.
          [ Interface changes ] Keep HMS interfaces backward compatible with changes for HIVE-21637 Sub-task Closed Kishen Das  
          5.
          [ Interface changes ] Add table id to HMS get methods Sub-task Resolved Kishen Das  
          6.
          [ Test-fix ] alter_table_update_status/alter_table_update_status_disable_bitvector/alter_partition_update_status fail when DbNotificationListener is installed Sub-task Open Unassigned  
          7.
          [RawStore] RawStore changes to facilitate HMS cache consistency Sub-task Open Unassigned  
          8.
          [CachedStore ] Prewarm HMS cache during bootstrap with ValidWriteIdList for all the tables Sub-task In Progress Ashish Sharma  
          9.
          [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper Sub-task In Progress Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4.5h
          10.
          [HMS] Make all Metastore reads consistent Sub-task Open Unassigned  
          11.
          [HMS] Advance the write id for the table for DDL Sub-task Closed Kishen Das  
          12.
          [CachedStore] Use notification log to keep the cache up-to-date with all the changes Sub-task Open Unassigned  
          13.
          [HMS Client ] During writes, mark writeId as committed locally in HMS client Sub-task Open Kishen Das  
          14.
          [CachedStore] Add check/default constraints in CachedStore Sub-task Closed Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          15.
          Send ValidWriteIDList in request for all the new HMS get_* APIs that are in request/response form Sub-task Closed Kishen Das  
          16.
          [CachedStore] Implement caching/fetching of foreign keys based on parent db/table Sub-task Open Adesh Kumar Rao  
          17.
          [HS2] Send tableId in request for get_table_request API Sub-task Closed Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h
          18.
          Send tableId in request for all the new HMS get_partition APIs Sub-task Resolved Kishen Das  
          19.
          [CachedStore] Add flag in TableWrapper in CacheStore to check if constraints are set or not Sub-task Closed Adesh Kumar Rao  
          20.
          Send ValidWriteIdList and tableId to get_*_constraints HMS APIs Sub-task Closed Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          21.
          [CachedStore] Data mismatch between CachedStore and ObjectStore for constraints Sub-task Closed Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          22.
          [CachedStore] Constraints read from cache should be consistent snapshot. Sub-task Closed Ashish Sharma

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 50m
          23.
          [cache store] Add valid flag in table wrapper for all constraint Sub-task Resolved Ashish Sharma  
          24.
          Advance write Id during AlterTableAddConstraint DDL Sub-task Resolved Kishen Das  
          25.
          Advance write Id during AlterTableDropConstraint DDL Sub-task Resolved Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          26.
          [HS2] Send tableId to get_partitions_by_names_req HMS API from HS2 Sub-task Resolved Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          27.
          [HMS] Populate tableId in the response of get_valid_write_ids API Sub-task Resolved Kishen Das  
          28.
          [HS2] Cache tableId in SessionState Sub-task Resolved Kishen Das  
          29.
          [HS2] Enhance DriverTxnHandler.isValidTxnListState logic to include tableId comparison Sub-task Resolved Kishen Das  
          30.
          [HMS] Provide new HMS API to return latest committed compaction record for a given table Sub-task Closed Yu-Wen Lai

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 40m
          31.
          Provide default implementation for HMS APIs Sub-task Closed Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          32.
          [cachedstore] Add tableId and writeId check in cachedStore Sub-task Open Ashish Sharma  
          33.
          [HMS] Advance write Id during AlterTableDropPartition Sub-task Closed Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          34.
          [Hive] Advance write ID for remaining DDLs Sub-task Resolved Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          35.
          Advance Write ID during ALTER TABLE ( NOT SKEWED, SKEWED BY, SET SKEWED LOCATION, UNSET SERDEPROPERTIES) Sub-task Closed Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          36.
          Future ALTER TABLE flows should advance write Id for transactional table Sub-task In Progress Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          37.
          Add a test case to ensure Truncate table advances the write ID Sub-task Closed Kishen Das

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          38.
          ValidWriteIdList & table id are sometimes missing when requesting partitions by name via HS2 Sub-task Closed Stamatis Zampetakis

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          39.
          Add unit tests for Hive#getPartitionsByNames using batching Sub-task Closed Stamatis Zampetakis

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m
          40.
          Drop unused requests from TestHiveMetaStoreClientApiArgumentsChecker Sub-task Closed Stamatis Zampetakis

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m

          Activity

            People

              kishendas Kishen Das
              daijy Daniel Dai
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 38h 10m
                  38h 10m