Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2756

Rethink UDF interfaces

    XMLWordPrintableJSON

    Details

      Description

      In the next update of the UDF interfaces, we want to make sure that Status is returned for certain functions such as FunctionContext::Allocate(), Reallocate() and may be Free().

      Currently, these functions can return a valid pointer to a buffer even if the memory limit has been exceeded. It relays the error status (e.g. memory limit exceeded) via the FunctionContext object and relies on various poll points in the code to call QueryMaintenance() to catch the problem asynchronously and abort the query.

      The long term goal would be to enforce the memory limit synchronously in these functions (i.e. essentially removing Allocate and replacing it with TryAllocate()) and get rid of all the poll loop of status check in QueryMaintenance() if possible. This would also give us a tighter bound on the memory consumption, potentially avoiding some corner cases in which impalad may overrun the process memory limit.

      Please see IMPALA-2620 and http://gerrit.cloudera.org:8080/#/c/1445 for discussions about this problem.

      Also, FunctionContext::Allocate(), Reallocate() should support 64-bit allocations so the parameter 'size' has to be 64-bit.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kwho Michael Ho
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: