[DRILL-5146] Unnecessary spilling to disk by sort when we only have 5000 rows with one column

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Execution - Relational Operators
• Assignee: Paul Rogers
• Reporter: Rahul Kumar Challapalli
• Created: 21/Dec/16 19:15
• Updated: 15/Sep/17 07:12
• Resolved: 17/Aug/17 03:31

Description

git.commit.id.abbrev=cf2b7c7

The below query spills to disk for the sort. The dataset contains 5000 files and each file contains a single record.

select * from dfs.`/drill/testdata/resource-manager/5000files/text` order by columns[1];

Enviironment :

DRILL_MAX_DIRECT_MEMORY="16G"
DRILL_MAX_HEAP="4G"

I attached the dataset, logs and the profile

Issue Links

• Is contained by: DRILL-5601 Rollup of External Sort memory management fixes