[IMPALA-4179] Remove RowBatch::MarkNeedsDeepCopy() memory management API

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: resource-management
• Epic Link: Simplify memory transfer model
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Tim Armstrong
• Created: 21/Sep/16 20:47
• Updated: 25/Feb/20 01:35

Description

We should remove this API as a simplification of the memory transfer model.

It was used in various places to manage lifetime of BufferHandles, but those can be replaced with attaching the BufferHandle + MarkFlushResources().

It is also used to work around the fact that some memory is never returned from ExecNodes and is freed in Close(). Part of the solution is probably to add a way to attach all resources to a RowBatch before Close() and during Reset().

Issue Links

• depends upon: IMPALA-5844 Fix management of FunctionContext "local" allocations.
• is blocked by: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
• relates to: IMPALA-9156 Share broadcast join builds between fragments
• relates to: IMPALA-3652 Fix resource transfer in subplans with limits

Sub-Tasks

• Remove MarkNeedsDeepCopy() from Sorter: Resolved
• Remove MarkNeedsDeepCopy from Aggregation and Hash Join Nodes: Resolved
• Remove MarkNeedsDeepCopy() from UnionNode: Open