Could we put the clearSSTableReadMeter in CFS or SSTR? Or put it in SSTableDeletingDask instead of making it a notification at all. Alternatively, since it's just one row I'd lean towards just letting TTL take care of it.
Going with SSTR would result in duplicate notifications. I think SSTableDeletingTask is a good spot, I just wasn't sure if that would be appropriate. If the TTL was short (say, less than 1 day), I think that would be okay, but...
Does TTL need to be so long if we're persisting every 5m?
I considered making it smaller, but in the case where a node goes down for some amount of time, it would be nice to not lose all of the stats when it comes back up.
It looks like having the increments in the read section of the iterators means we only increment in index lookup (getPosition) is successful. IMO we should increment before getPosition. May be cleaner to do this in collationController but iterator constructor also works.
Just to be check, getPosition() failure indicates a BF false-positive, but we want to include those in the count? I can see this making sense for managing in-memory index summary sizes, but maybe not for STCS optimization. (It should be a small enough number not to matter much either way.)
What can we do to restore coldness data from a snapshot?
Not much with the current storage strategy. Do we have any workarounds/suggestions for restoring TTLed data in general?