This patch addresses a TODO in QueryRescorer: We should not sort the full array of the results returned from rescoring, but rather only topN, when topN is less than total hits.
Made this optimization with some suggestions from jpountz and jimczi, this is my first lucene patch submission.