[SOLR-15859] Add handler to dump filter cache

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: FQ cache filtercache metrics
• Assignee: Shawn Heisey
• Reporter: Andy Lester
• Created: 20/Dec/21 18:05
• Updated: 22/Dec/22 05:48

Description

It would be very helpful to be able to inspect the contents of the filterCache.

I'd like to be able to query something like

/admin/caches?type=filter&nentries=1000&sort=numHits+DESC

nentries would be allowed to be -1 to get everything.

It would be nice to see these data items for each entry. I don't know which are available, but I'm thinking blue sky here:

• cache key, exactly as stored
• Timestamp when the entry was inserted
• Whether the insertion of the entry evicted another entry, and if so which one
• Timestamp of when this entry was last hit
• Number of hits on this entry forever
• Number of hits on this entry over some time period
• Number of documents matched by the filter
• Number of bytes of memory used by the filter

These are the sorts of questions I'd like to be able answer:

• "I just did a query that I expect will have added a cache entry. Did it?"
• "Are my queries hitting existing cache entries?"
• "How big should I set my filterCache size? Should I limit it by number of entries or RAM usage?"
• "Which of my FQs are getting used the most? These are the ones I want in my firstSearcher queries." (I currently determine this by processing my old solr logs)
• "Which filters give me the most bang for the buck in terms of RAM usage?"
• "I have filter X and filter Y, but would it be beneficial if I made a filter X AND Y?"
• "Which FQs are used more at certain times of the day? (Assuming I take regular snapshots throughout the day)"

I imagine a response might look like:

{{{}}
  "responseHeader": {
    "status": 0,
    "QTime": 961
  },
  "response": {
    "numFound": 12104,
    "filterCacheKeys": {
      [
        "language:eng": {
          "inserted": "2021-12-04T07:34:16Z",
          "lastHit": "2021-12-04T18:17:43Z",
          "numHits": 15065,
          "numHitsInPastHour": 2319,
          "evictedKey": "agelevel:4 shippable:Y",
          "numRecordsMatchedByFilter": 24328753,
          "bytesUsed": 3041094
{{        }}}
      ],
      [
        "is_set:N": {
          ...
{{        }}}
      ],
      [
        "language:spa": {
          ...
{{        }}}
      ]
{{    }}}
}

Issue Links

• split to: SOLR-16548 CaffeineCache seems to be not working in 9.0 and later