[IMPALA-2756] Rethink UDF interfaces

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.3.0
• Component/s: Backend
• Labels: incompatibility resource-management
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Michael Ho
• Created: 09/Dec/15 00:25
• Updated: 23/Oct/18 22:08

Description

In the next update of the UDF interfaces, we want to make sure that Status is returned for certain functions such as FunctionContext::Allocate(), Reallocate() and may be Free().

Currently, these functions can return a valid pointer to a buffer even if the memory limit has been exceeded. It relays the error status (e.g. memory limit exceeded) via the FunctionContext object and relies on various poll points in the code to call QueryMaintenance() to catch the problem asynchronously and abort the query.

The long term goal would be to enforce the memory limit synchronously in these functions (i.e. essentially removing Allocate and replacing it with TryAllocate()) and get rid of all the poll loop of status check in QueryMaintenance() if possible. This would also give us a tighter bound on the memory consumption, potentially avoiding some corner cases in which impalad may overrun the process memory limit.

Please see IMPALA-2620 and http://gerrit.cloudera.org:8080/#/c/1445 for discussions about this problem.

Also, FunctionContext::Allocate(), Reallocate() should support 64-bit allocations so the parameter 'size' has to be 64-bit.

Sub-Tasks

• FunctionContext::*Allocate*() should use TryAllocate() instead of Allocate().: Open