[ARROW-14202] [C++] A more RAM-efficient top-k sink node

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 7.0.0
• Component/s: C++
• Labels: query-engine
• External issue URL: https://github.com/apache/arrow/issues/29786
• Assignee: Unassigned
• Reporter: Alexander Ocsa
• Created: 01/Oct/21 20:25
• Updated: 11/Jan/23 08:38

Description

Mentioned here:

https://github.com/apache/arrow/pull/11274#pullrequestreview-768267959

For example, a top-k implementation could periodically (when batches_ has some configurable # of rows) run through and discard data. The way it is written now it would still require me to buffer the entire dataset in memory (and/or spillover).

 

 

Issue Links

• is depended upon by: ARROW-14254 [C++] Return a random sample of rows from a query
• is duplicated by: ARROW-14201 RAM-efficient topk sink node