Currently it is not possible to use RankQuery  and Grouping  together (see also ). In some situations Grouping can be replaced by Collapse and Expand Results  (that supports reranking), but i) collapse cannot guarantee that at least a minimum number of groups will be returned for a query, and ii) in the Solr Cloud setting you will have constraints on how to partition the documents among the shards.
I'm going to start working on supporting RankQuery in grouping. I'll start attaching a patch with a test that fails because grouping does not support the rank query and then I'll try to fix the problem, starting from the non distributed setting (GroupingSearch).
My feeling is that since grouping is mostly performed by Lucene, RankQuery should be refactored and moved (or partially moved) there.
Any feedback is welcome.