[DRILL-5616] Hash Agg Spill: OOM while reading irregular varchar data

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Execution - Relational Operators
• Labels: ready-to-commit
• Assignee: Boaz Ben-Zvi
• Reporter: Boaz Ben-Zvi
• Reviewer: Paul Rogers
• Created: 28/Jun/17 22:31
• Updated: 21/Jul/17 17:35
• Resolved: 21/Jul/17 17:35

Description

An OOM while aggregating a table of two varchar columns where sizes vary significantly ( about 8 bytes long in average, but 250 bytes max )

alter session set `planner.width.max_per_node` = 1;
alter session set `planner.memory.max_query_memory_per_node` = 327127360;
select count( * ) from (select max(`filename`) from dfs.`/drill/testdata/hash-agg/data2` group by no_nulls_col, nulls_col) d;

Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

OOM at Second Phase. Partitions: 2. Estimated batch size: 12255232. Planned batches: 0. Rows spilled so far: 434127447 Memory limit: 163563680 so far allocated: 150601728.
Fragment 1:0

Issue Links

• links to: GitHub Pull Request #871