Currently we have 4 types of grouping collectors and 8 concrete subclasses in Lucene / Solr. In current architecture for each type of collector two concrete subclasses need to be created. An implementation optimized for single term based groups and a more general implementation that works with MutableValue to also support grouping by functions. If we want for example group by IndexDocValues each type of grouping collector needs to have three concrete subclasses. This design isn't very maintainable.
I think it is best to introduce a concept that knows how deals with dealing groups for all the different sources. Therefore the grouping module should depend on the queries module, so that grouping can reuse the ValueSource concept. A term based concrete impl. of this concept knows for example to use the DocValues.ord() method. Or more generic concrete impl. will use DocValues.ValueFiller.