SortedSource or rather its impls. are hairy and horribly inefficient during merge.
Well let's address that ("remove SortedSource") under a different issue? I don't think we should remove them here because it's unrelated (there are other great simplifications here).
I'm all for simplifying, but not if we lose good functionality (cut into the bone).
Loading everything into heap can be solved, but, I think is relatively low priority because that's how the search app would use sorted DV at search time anyway. Ie, an app sets this on their "country" field because they know they want to sort by "country" at search time, and they will load the DV into RAM at search time. We only hold this RAM allocated while we merge the one field, then it's freed, so if multiple DV sort fields are used, merging takes less RAM than searching. I think the merge RAM usage is relatively minor.
The SortedSource has a simple API (just like FC.DocTermsIndex): getOrd(int docID), getValue(int ord). I don't see why this is adding complexity to DV.