Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8690

Better eviction algorithm for data cache

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 3.3.0
    • Fix Version/s: Impala 4.0
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-9

      Description

      With the current implementation of data cache, all data access will be cached regardless of the access pattern. The current LRU eviction algorithm is not resistant to scan traffic so in case some users scan a big fact table, a lot of the heavily accessed items will be evicted inevitably. We should adopt better eviction algorithm (e.g. LRFU or some other well known ones in the literature). Would be nice to evaluate it against some users' traces now that IMPALA-8542 is fixed.

      In the short run, we probably need some workaround (e.g. query hints to disable caching for certain tables). Will file a separate jira for it.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                kwho Michael Ho
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: