Normal Description
When we write the key cache, we write the RowIndeEntry entirely, as can be seen in CacheService.KeyCacheSerializer.serialize:
ByteBufferUtil.writeWithLength(key.key, out); out.writeInt(key.desc.generation); out.writeBoolean(true); key.desc.getFormat().getIndexSerializer(cfm, key.desc.version, SerializationHeader.forKeyCache(cfm)).serialize(entry, out);
When we deserialize such entry, we have the case where an entry correspon to a sstable that doesn't exist anymore (presumably, it's been deleted between the last cache saving and the restart that triggers the cache loading), which goes this way:
ByteBuffer key = ByteBufferUtil.read(input, keyLength);
int generation = input.readInt();
SSTableReader reader = findDesc(generation, cfs.getSSTables(SSTableSet.CANONICAL));
input.readBoolean(); // backwards compatibility for "promoted indexes" boolean
if (reader == null)
{
RowIndexEntry.Serializer.skipPromotedIndex(input);
return null;
}
But the thing is, RowIndexEntry.Serializer.skipPromotedIndex doesn't skip a full RowIndexEntry, it doesn't skip the position in particular, and so as far as I can tell, that part is buggy. I believe we should call RowIndexEntry.Serializer.skip.
I'll note that I just noticed this while reading the code but I haven't reproduced that problem personally. I also haven't pin-pointed the exact source of that problem, but it's been there for a while as far as I can tell. At the same time, the condition to reach that branch is probably pretty uncommon, and failure to load the key cache does not prevent the node from starting and it doesn't even seem that we log an error (we log at DEBUG, maybe we should log at WARN), which would explain nobody reporting this.