We have a case on a SolrCloud cluster. Queries takes too much QTime, due to a randomly slow shard request. In a noticeable part of queries, the slowest shard consumes more than 4 times qtime than the average.
Of course, deep inspection of the performance factor should be made on the specific environment.
But, there is one more idea:
If shard request will be sent to all of the replicas of each shard, the probability of all the replicas of the same shard to be the slowest is very small. Obviously cluster works harder, but on a (very) low qps, it might be OK.