The problem is you need more information than simply "these segments got merged" to actually do something interesting with your caches.
Okay, now I've thought a bit. What we need is a notification on which segments remained, which are new and which got toasted, plus docid ranges for them. Their ancestry is irrelevant, because you're right, to exploit it we also need deleted docs, and then replicate some of the merging logic and it gets really messy from here. Dropping parts of the cache related to dead segments, rebasing survivors and doing a fair-and-square load/uninversion/whatever for new ones is enough.
Can you explain what's missing in Lucene's FieldCache?
It's not that easy to say. Our version was initially used only for sorting, but without concurrency issues and with async warmup. But then we used it to load docs (way better than storing fields and using IndexReader.document), tied up with our strongly-typed-fields code, added handling for multi-valued fields, used it for faceted searches.
So now it is essentially just something different from Lucene field cache.