Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21697

Remove periodical full refresh in HMS cache

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Standalone Metastore
    • Labels:
      None
    • Target Version/s:

      Description

      In HIVE-18661, we added periodical notification based refresh in HMS cache. We shall remove periodical full refresh to simplify the code as it will no longer be used. In the mean time, we introduced mechanism to provide monotonic reads through the CachedStore.commitTransaction. This will no longer be needed after HIVE-21637. So I will remove related code as well. This will provide some performance benefits include:
      1. We don't have to slow down write to catch up notification logs. Write can be done immediately and tag the cache with writeids
      2. We can read from cache even if updateUsingNotificationEvents is running. Read will compare the writeids of the cache so monotonic reads will be guaranteed

      I'd like to put a patch separately with HIVE-21637 so it can be tested independently. HMW will use periodical notification based refresh to update cache. And it will temporarily lift the monotonic reads guarantee until HIVE-21637 checkin.

        Attachments

        1. HIVE-21697.8.patch
          163 kB
          Jianyong Dai
        2. HIVE-21697.7.patch
          133 kB
          Jianyong Dai
        3. HIVE-21697.6.patch
          133 kB
          Jianyong Dai
        4. HIVE-21697.5.patch
          132 kB
          Jianyong Dai
        5. HIVE-21697.4.patch
          130 kB
          Jianyong Dai
        6. HIVE-21697.3.patch
          84 kB
          Jianyong Dai
        7. HIVE-21697.2.patch
          84 kB
          Jianyong Dai
        8. HIVE-21697.1.patch
          71 kB
          Jianyong Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Jianyong Dai
                Reporter:
                daijy Jianyong Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: