[DRILL-5154] OOM error in external sort on top of 400GB data set generated using terasort benchamark

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.10.0
• Component/s: Execution - Relational Operators
• Assignee: Paul Rogers
• Reporter: Rahul Kumar Challapalli
• Created: 22/Dec/16 19:23
• Updated: 15/Sep/17 06:43
• Resolved: 15/Sep/17 06:43

Description

git.commit.id.abbrev=cf2b7c7

The below query fails with an OOM in external sort

No of drillbits : 1
Nodes in Mapr cluster : 2
DRILL_MAX_DIRECT_MEMORY="16G"
DRILL_MAX_HEAP="4G"
select * from (select * from dfs.`/drill/testdata/resource-manager/terasort-data/part-m-00000.tbl` order by columns[0]) d where d.columns[0] = 'null';
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Unable to allocate buffer of size 8388608 due to memory limit. Current allocation: 8441872
Fragment 1:6

[Error Id: 87ede736-b480-4286-b472-7694fdd2f7da on qa-node183.qa.lab:31010] (state=,code=0)

I attached the logs and the query profile