[IMPALA-5844] Fix management of FunctionContext "local" allocations.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Epic Link: Simplify memory transfer model
• Target Version: Impala 2.11.0
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 24/Aug/17 20:48
• Updated: 10/Aug/18 16:36
• Resolved: 06/Oct/17 00:02

Description

FunctionContexts associated with expressions make two kind of allocations with very different lifecycles. One type of allocation is owned and managed by the expression, while the other "local" allocation is implicitly transferred to the Impala daemon after control flow returns from the expression. Both are currently allocated from the same pool.

RowBatches returned from plan nodes may reference variable-length data stored in local allocations so this memory should be attached to the RowBatches.

One approach here is:

• Separate local and other allocations to allocate from different MemPools.
• Manage local allocations in bulk by clearing, freeing, or transferring data from that MemPool, similar to other memory that would be allocated from a MemPool.

I think there are some potential wrinkles here and work to set up correct MemPools for all places that use expressions but I think the high-level approach should work.

Issue Links

• is depended upon by: IMPALA-4179 Remove RowBatch::MarkNeedsDeepCopy() memory management API
• is depended upon by: IMPALA-2399 Cleanup/rethink QueryMaintenance() calls in the BE.
• is related to: IMPALA-5911 Grouping aggregations with having conjuncts and Serialize()/Finalize() functions uses excessive expr memory