[DRILL-5284] Roll-up of final fixes for managed sort

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.11.0
• Labels: ready-to-commit
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Reviewer: Boaz Ben-Zvi
• Created: 21/Feb/17 19:13
• Updated: 28/Mar/17 20:18
• Resolved: 02/Mar/17 22:49

Description

The managed external sort was introduced in DRILL-5080. Since that time, extensive testing has identified a number of minor fixes and improvements. Given the long PR cycles, it is not practical to spend a week or two to do a PR for each fix individually. This ticket represents a roll-up of a combination of a number of fixes. Small fixes are listed here, larger items appear as sub-tasks.

Issue Links

• links to: GitHub Pull Request #761

Sub-Tasks

• Provide detailed, accurate estimate of size consumed by a record batch: Resolved
• Managed external sort spills too often with Parquet data: Closed
• Managed External Sort fails with OOM: Closed
• External sort refers to the deprecated HDFS fs.default.name param: Resolved
• Config param drill.exec.sort.external.batch.size is not used: Closed
• NPE in managed external sort while spilling to disk: Closed
• OOM : Unable to allocate sv2 for 1000 records, and not enough batchGroups to spill.: Resolved
• External Sort BatchGroup leaks memory if an OOM occurs during read: Resolved
• Managed External Sort throws an OOM during the merge and spill phase: Closed