[DRILL-1942] Improve off-heap memory usage tracking

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: Future
• Component/s: Execution - Relational Operators
• Assignee: Chris Westin
• Reporter: Chris Westin
• Created: 06/Jan/15 23:24
• Updated: 24/Jun/16 21:50

Description

We're using a lot more memory than we think we should. We may be leaking it, or not releasing it as soon as we could.

This is a call to come up with some improved tracking so that we can get statistics out about exactly where we're using it, and whether or not we can release it earlier.

Issue Links

• depends upon: DRILL-3920 Add vector loading tests
• depends upon: DRILL-3927 Use OutOfMemoryException in more places
• depends upon: DRILL-3930 Remove direct references to TopLevelAllocator from unit tests
• depends upon: DRILL-3940 Make RecordBatch AutoCloseable
• incorporates: DRILL-2421 ensure all allocators for a query are descendants of a single root
• incorporates: DRILL-3033 Add memory leak fixes found so far in DRILL-1942 to 1.0
• incorporates: DRILL-3040 Accountor drill.exec.memory.enable_frag_limit not defaulted normally
• incorporates: DRILL-3594 RuleSets capture and illegally use allocator from the first query executed on the drillbit
• incorporates: DRILL-3598 Use a factory to create the root allocator
• is depended upon by: DRILL-3616 Memory leak in a cleanup code after canceling queries with window functions spilling to disk
• is depended upon by: DRILL-2515 Add allocator state verification at the end of test suites
• is depended upon by: DRILL-2757 Verify operators correctly handle low memory conditions and cancellations
• is depended upon by: DRILL-2698 Add allocation policy for (direct memory) limits per-query-per-node
• is depended upon by: DRILL-1992 Add more stats for HashJoinBatch and HashAggBatch
• relates to: DRILL-3064 TestSimpleExternalSort is completely @Ignored
• relates to: DRILL-2219 Concurrent modification exception in TopLevelAllocator if a child allocator is added during loop in close()
• relates to: DRILL-2719 Behavior of ValueVector#getBuffers(clear) is inconsistent across VV types
• relates to: DRILL-2847 DrillBufs from the RPC layer are being leaked

Sub-Tasks

• Memory leak in a cleanup code after canceling queries with window functions spilling to disk: Closed