[DRILL-5325] Implement sub-operator unit tests for managed external sort

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Tools, Build & Test
• Assignee: Paul Rogers
• Reporter: Paul Rogers
• Reviewer: Boaz Ben-Zvi
• Created: 07/Mar/17 18:38
• Updated: 22/May/18 17:12
• Resolved: 08/Jul/17 01:03

Description

Validate the proposed sub-operator test framework, by creating low-level unit tests for the managed version of the external sort.

The external sort has a small number of existing tests, but those tests are quite superficial; the "managed sort" project found many bugs. The managed sort itself was tested with ad-hoc system-level tests created using the new "cluster fixture" framework. But, again, such tests could not reach deep inside the sort code to exercise very specific conditions.

As a result, we spent far too much time using QA functional tests to identify specific code issues.

Using the sub-opeator unit test framework, we can instead test each bit of functionality at the unit test level.

If doing so works, and is practical, it can serve as a model for other operator testing projects.

Issue Links

• causes: DRILL-6435 MappingSet is stateful, so it can't be shared between threads
• contains: DRILL-5163 External sort on Mac creates a separate child process per spill via HDFS FS
• links to: GitHub Pull Request #808

Sub-Tasks

• Memory leak in managed sort if OOM during sv2 allocation: In Progress
• External sort fails with OOM error (Fails to allocate sv2): Closed
• Refactor "managed" external sort for unit tests: Resolved