The additions to VIntCoding are now unused
Moving the offsets buffer to the beginning is a mistake I think unless you plan to reallocate all the time. I don't think you should. When we are building these they are for new sstables that are likely to be compacted away so the time spent compacting the RIE is not productive? Also the sizing is off, the offsets are much smaller so writing to the offsets DataOutputBuffer will require more doubling/reallocation. I am not sure if it will reallocate to the correct size efficiently. If you are going through all that and you know the serialized size of everything anyways it seems like a waste just allocate the correct size BB manually and put everything in it.
When deserializing an old format IndexedEntry I think you have to rewrite it to generate the offsets. Otherwise the cache will never be populated with an entry where the offsets are calculated. That will make it slower than the older version. I also think this will be faster than generating it incrementally since it's a nice tight loop doing a scan of memory. If you are doing a binary search you will end up doing most of that work anyways and if it's a scan you will also end up doing most of it.
RowIndexEntry.java line 423, legacyIndexInfoSearch. It's still doing a loop from the beginning of the offsets to the last offset it calculated. There is no need we should know the last calculated offset available and skip to it. For a scan this operation becomes n^2 with that loop. I think it should go away completely. Just rewrite the IndexedEntry during deserialization since you are making a copy anyways when you bring it out of the field.
Jonathan told me the expectation is that people run upgrade sstables so we don't need to be heroic. Let's go for the simples possible solution which is making the old and new formats match after deserialization. Hopefully this means we can remove a bunch of paths based in which format we are looking at.
For cache hits we have to copy the entire IndexedEntry onto the heap into unpooled memory. That is making an operation that was lg N a linear operation to the size of the IndexedEntry. In terms of raw speed the on heap cache is going to be better off using the new serialization, but it will really poke the garbage collector in the eye. At least with the OHC cache the garbage is short lived.
I don't like to give people options they have to choose from, but I am more afraid of making the product unworkable for some use case. Maybe we should allow the key cache to be selectable for 3.0? I feel like these are the two options that get us to 3.0 while minimizing regret post release.
We have one week so it's less is more time.