Here is a patch. It splits the FieldComparator API into FieldComparator and LeafFieldComparator (like Collector and LeafCollector).
As a consequence our TopDocsCollectors do not extend SimpleCollector anymore and return a new LeafCollector instance per leaf.
TopFieldCollector was a bit of a pain to migrate because of all the specialization, so I replaced the OneComparator/MultiComparator specialization with some wrapping of the LeafFieldComparator API so that it always seems like there is a single comparator to the TopFieldCollector (only the leaf comparators are wrapped, not the top-level ones so that the situation is not weird because of the value(int slot) method that can return a single value). This helped remove a significant amount of code (7 collectors instead of 13).
Some grouping/join collectors were a bit hard to migrate so for now they still extend SimpleCollector and keep a reference to the currently wrapped leaf comparator/collector. I think this can be addressed later?
For now the patch only changes the API but we could imagine doing funny things now that the top docs collectors and comparators are fully per-segment, like only counting hits on segments that are not competitive (because of their min/max value or because they don't have a value for a given field – if we removed the constraint that TopDocsCollector needs to return the total hit count, we could even completely skip such segments) or not worrying about the reader generation of the first segment when using ordinals to sort.