[DRILL-5755] TOP_N_SORT operator does not free memory while running

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Execution - Relational Operators
• Assignee: Timothy Farkas
• Reporter: Boaz Ben-Zvi
• Reviewer: Paul Rogers
• Created: 30/Aug/17 22:59
• Updated: 04/Oct/17 09:47
• Resolved: 04/Oct/17 09:47

Description

The TOP_N_SORT operator should keep the top N rows while processing its input, and free the memory used to hold all rows below the top N.

For example, the following query uses a table with 125M rows:

select row_count, sum(row_count), avg(double_field), max(double_rand), count(float_rand) from dfs.`/data/tmp` group by row_count order by row_count limit 30;

And failed with an OOM when each of the 3 TOP_N_SORT operators was holding about 2.44 GB !! (see attached profile). It should take far less memory to hold 30 rows !!

Issue Links

• relates to: DRILL-5138 TopN operator on top of ~110 GB data set is very slow
• links to: GitHub Pull Request #964