This issue occurs when using the grouping feature in distributed mode and sorting by score.
Each group's docList in the response is supposed to contain a maxScore entry that hold the maximum score for that group. Using the current releases, it sometimes happens that this piece of information is not included:
Looking into the issue, it comes from the fact that if a shard does not contain a document from that group, trying to merge its maxScore with real maxScore entries from other shards is invalid (it results in NaN).
I'm attaching a patch containing a fix.