Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7127 Fetch-on-demand metadata for the impalad-side catalog
  3. IMPALA-7534

Handle invalidation races in CatalogdMetaProvider cache

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Not Applicable
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-3

      Description

      There is a well-known race in Guava's LoadingCache that we are using for CatalogdMetaProvider which we are not currently handling:

      • thread 1 gets a cache miss and makes a request to fetch some data from the catalogd. It fetches the catalog object with version 1 and then gets context switched out or otherwise slow
      • thread 2 receives an invalidation for the same object, because it has changed to v2. It calls 'invalidate' on the cache, but nothing is yet cached.
      • thread 1 puts back v1 of the object into the cache

      In essence we've "missed" an invalidation. This is also described in this nice post: https://softwaremill.com/race-condition-cache-guava-caffeine/

      The race is quite unlikely but could cause some unexpected results that are hard to reason about, so we should look into a fix.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: