Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
None
-
None
-
None
-
None
Description
When, for example, doing an order by with a limit, if limit << total, it would be much more efficient to maintain a priority queue instead of sorting the entire data set.
In most cases, this will greatly reduce the number of comparisons, since most incoming records will not fall in the Top N, and thus will only require a single comparison operation. Incoming records that are in the Top-N will require at most log N comparisons.
This will also allow periodic purging of record batches, reducing memory requirements.