Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-15635

[C++][Python] UDF Integration



    • Task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • C++, Python


      The objective is to list down a set of tasks required to provide UDF support for Apache Arrow streaming execution engine. In the first iteration we will be focusing on providing support for Python-based UDFs which can support Python functions. 

      The UDF Integration is going to pan out with a series of sub-tasks associated with the development and PoCs. Note that this is going to be the first iteration of UDF integrations with a limited scope. This ticket will cover the following topics;

      1. POC for UDF integration: The objective is to evaluate the existing components in the source and evaluate the required modifications and new building blocks required to integrate UDFs.
      2. The language will be limited to C+/Python users can register Python function as a UDF and use it with an `apply` method on Arrow Tables or provide a computation API endpoint via arrow::compute API. Note that the C+ API already provides a way to register custom functions via the function registry API. At the moment this is not exposed to Python. 
      3. Planned features for this ticket are;
        1. Scalar UDFs : UDFs executed per value (per row)
        2. Vector UDFs : UDFs executed per batch (a full array or partial array)
        3. Aggregate UDFs : UDFs associated with an aggregation operation
      4. Integration limitations
        1. Doesn't support custom data types which doesn't support Numpy or Pandas
        2. Complex processing with parallelism within UDFs are not supported
        3. Parallel UDFs are not supported in the initial version of UDFs. Allthough we are documenting what is required and a rough sketch for the next phase. 


        Issue Links



              vibhatha Vibhatha Lakmal Abeykoon
              vibhatha Vibhatha Lakmal Abeykoon
              0 Vote for this issue
              6 Start watching this issue



                Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0h
                  Time Spent - 77h 50m
                  77h 50m