[DRILL-2757] Verify operators correctly handle low memory conditions and cancellations

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 1.0.0
• Component/s: Execution - Flow, Execution - Relational Operators
• Assignee: Steven Phillips
• Reporter: Chris Westin
• Created: 10/Apr/15 20:34
• Updated: 22/May/15 23:08
• Resolved: 10/May/15 16:24

Description

Check that the path through sort that notices low memory conditions and causes the sort to spill ("out of memory condition management").

Also check to make sure we handle queries and fragment failures properly under these conditions.

"hashjoin, hashagg, and topn use large amounts of memory,and may be unable to
complete if their memory needs can't be met; for all others, the idea is that they can complete if they get their reservation"

Issue Links

• contains: DRILL-2947 AllocationHelper.allocateNew() doesn't have a consistent behavior when it can't allocate
• contains: DRILL-2893 ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY
• contains: DRILL-2895 AbstractRecordBatch.buildSchema() should properly handle OUT_OF_MEMORY outcome
• contains: DRILL-2905 RootExec implementations should properly handle IterOutcome.OUT_OF_MEMORY
• contains: DRILL-2920 properly handle OutOfMemoryException
• contains: DRILL-2894 FixedValueVectors shouldn't set it's data buffer to null when it fails to allocate it
• contains: DRILL-2816 system error does not display the original Exception message
• depends upon: DRILL-2756 Check for cancellation in pseudo-blocking operations
• depends upon: DRILL-1942 Improve off-heap memory usage tracking
• is depended upon by: DRILL-2878 FragmentExecutor.closeOutResources() is not called if an exception happens in the Foreman before the fragment executor starts running
• relates to: DRILL-3064 TestSimpleExternalSort is completely @Ignored
• relates to: DRILL-2314 Drill on windows fail to make spill diretory for queries against large data

Sub-Tasks

• ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY: Resolved
• AbstractRecordBatch.buildSchema() should properly handle OUT_OF_MEMORY outcome: Resolved
• RootExec implementations should properly handle IterOutcome.OUT_OF_MEMORY: Resolved
• properly handle OutOfMemoryException: Resolved
• AllocationHelper.allocateNew() doesn't have a consistent behavior when it can't allocate: Resolved