For partitions < 64k (partitions without an IndexInfo object) we could skip the indirection during reads via RowIndexEntry at all by extending the IndexSummary and directly store the offset into the data file
Since the idea here is to do something simple that we can be confident about shipping in 3.6 if CASSANDRA-9754 isn't ready, let's avoid making changes to the on disk layout.
To clarify for others following along,
Remove IndexInfo from the key cache (not from the index file on disk, of course)
This sounds scary but it's core to the goal here: if we're going to support large partitions, we can't afford the overhead either of keeping the entire summary on heap, or of reading it from disk in the first place. (If we're reading a 1KB row, then reading 2MB of summary first on a cache miss is a huge overhead.) Moving the key cache off heap (
CASSANDRA-9738) would have helped with the first but not the second.
So one approach is to go back to the old strategy of only caching the partition key location, and then go through the index bsearch using the offsets map every time. For small partitions this will be trivial and I hope negligible to the performance story vs the current cache. (If not, we can look at a hybrid strategy, but I'd like to avoid that complexity if possible.)
what I was thinking was that the key cache instead of storing a copy of the RIE it would store an offset into the index that is the location of the RIE. Then the RIE could be accessed off heap via a memory mapping without doing any allocations or copies
I was thinking that even the offsets alone for a 4GB partition are going to be 256KB, so we don't want to cache the entire offsets map. But the other side there is that if you have a bunch of 4GB partitions you won't have very many of them. 16TB of data would be 1GB of offsets which is within the bounds of reasonable when off heap. And your approach may require less logic changes than the one above, since we're still "caching" the entire summary, sort of; only adding an extra indirection to read the IndexInfo entries. So that might well be simpler.
Edit: but switching to a per-row cache (from per-partition) would be a much bigger change and I don't see the performance implications as straightforward at all, so let's not do that.