[DRILL-5080] Create a memory-managed version of the External Sort operator

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.11.0
• Labels: ready-to-commit
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Reviewer: Rahul Kumar Challapalli
• Created: 28/Nov/16 17:57
• Updated: 28/Mar/17 22:43
• Resolved: 13/Feb/17 16:18

Description

We propose to create a "managed" version of the external sort operator that works to a clearly-defined memory limit. Attached is a design specification for the work.

The project will include fixing a number of bugs related to the external sort, include as sub-tasks of this umbrella task.

Issue Links

• links to: GitHub Pull Request #717

Sub-Tasks

• External sort attempts to retry sv2 memory alloc, even if can never succeed: Resolved
• Refactor, document and simplify ExternalSortBatch: Resolved
• ExternalSortBatch cache size, spill count differs from config setting: Resolved
• ExternalSortBatch has inconsistent notions of the memory limit: Resolved
• ExternalSortBatch does not spill fully, throws off spill calculations: Resolved
• ExternalSortBatch sets two different limits for "copier" memory: Resolved
• ExternalSortBatch uses two memory allocators; one will do: Resolved
• External Sort Batch memory use depends on record width: Resolved
• ExternalSortBatch is inefficient: rewrites data unnecessarily: Resolved
• ExternalSortBatch spills all batches to disk even if even one spills: Resolved
• External Sort does not delete spill file if error occurs during close: Resolved
• ExternalSortBatch provides weak control over spill file size: Resolved
• External Sort pop shadows variable "initialAllocation" from AbstractBase: Resolved
• When sort runs out of memory and query fails, resources are seemingly not freed: Closed