Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7127 Fetch-on-demand metadata for the impalad-side catalog
  3. IMPALA-7534

Handle invalidation races in CatalogdMetaProvider cache

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.3.0
    • None
    • None
    • ghx-label-3

    Description

      There is a well-known race in Guava's LoadingCache that we are using for CatalogdMetaProvider which we are not currently handling:

      • thread 1 gets a cache miss and makes a request to fetch some data from the catalogd. It fetches the catalog object with version 1 and then gets context switched out or otherwise slow
      • thread 2 receives an invalidation for the same object, because it has changed to v2. It calls 'invalidate' on the cache, but nothing is yet cached.
      • thread 1 puts back v1 of the object into the cache

      In essence we've "missed" an invalidation. This is also described in this nice post: https://softwaremill.com/race-condition-cache-guava-caffeine/

      The race is quite unlikely but could cause some unexpected results that are hard to reason about, so we should look into a fix.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment