Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2895 Refine query cache
  3. KYLIN-2894

Change the query cache expiration strategy by signature checking

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: None
    • Fix Version/s: v2.6.0
    • Component/s: Query Engine
    • Labels:
      None

      Description

      Currently to invalid query cache, CacheService will either invoke cleanDataCache or cleanAllDataCache. Both methods will clear all of the query cache, which is very inefficient. In eBay PROD environment, there's around 400 cubing jobs per day, which means the query cache will be cleared very 4 minutes. Then we introduced a signature based cache invalidation strategy. The basic idea is as follows:

      • Add a signature for SQLResponse, here we choose the cube last build time
      • When fetch SQLResponse for cache, first check whether the signature is consistent. If not, this cached value is overdue and will be invalidate.

        Attachments

          Activity

            People

            • Assignee:
              Wang Ken Wang Ken
              Reporter:
              yaho Zhong Yanghong
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: