[DRILL-2728] Merge spill files in ExternalSort

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: Execution - Relational Operators
• Assignee: Steven Phillips
• Reporter: Steven Phillips
• Created: 09/Apr/15 01:10
• Updated: 15/Apr/15 08:56
• Resolved: 15/Apr/15 08:56

Description

When external sort spills, it first merges all of the in-memory batches, and then keeps one batch in memory, while spilling the rest to disk. It's possible that we will still run out of memory, due to the single batch held in memory for each spill. When the memory due to these batches becomes too large, we should combine the spill files.