There's one big flaw in this which I've now realized: upserts
The issue is that, even after we upsert a KV and drop its reference from the KV set, we don't actually free any memory, since that KV's byte was just a pointer to a larger chunk. So assumedly there is at least one other KV pointing into that same chunk. Let's say a 2M chunk supports 20000 incrementable cells. If we increment 19999 of them, we still have one KV pointing to it which uses up 2MB of RAM.
Not sure exactly how to solve this... will sleep on it a few days...
One thought is that the Memstore can keep an IdentityHashMap<byte, AtomicInteger> chunk->referred_size map around. When we upsert something (and hence remove its KV from kvset) we decrement the value in this map for the chunk it referred to. If the amount of referred-to data in a chunk falls below some threshold (say 50%) then we run some "compaction" code which relocates the remaining values to a new contiguous chunk. Performance impact of this might be too much, but maybe addressable with striping the counters or something.