Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.1-incubating
-
None
-
None
Description
Sort operator consumes all records before producing sorted records. For certain use cases where only Top N records are required, today sort consumes all records into memory and overflows( spills ) to disk. This impacts performance.
if topN is pushed down to sort, only required memory can be allocated and sort would only hold topN records in memory. Once all the records are read, sorted records in topN is returned.
