Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8894

[C++] C++ array kernels framework and execution buildout (umbrella issue)

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • C++
    • None

    Description

      In the wake of ARROW-8792, this issue is to serve as an umbrella issue for follow up work and associated "buildout" which includes things like:

      • Implementation of many new function types and adding new kernel cases to existing functions
      • Adding implicit casting functionality to function execution
      • Creation of "bound" physical array expressions and execution thereof
      • Pipeline execution (executing multiple kernels while eliminating temporary allocation)
      • Parallel execution of scalar and aggregate kernels (including parallel execution of pipelined kernels)

      There's quite a few existing JIRAs in the project that I'll attach to this issue and I'll open plenty more issues as things occur to me to help organize the work.

      Attachments

        Issue Links

          is a parent of

          New Feature - A new feature of the product, which has yet to be developed. ARROW-555 [C++] String algorithm library for StringArray/BinaryArray

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1489 [C++] Add casting option to set unsafe casts to null rather than some garbage value

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1574 [C++] Implement kernel function that converts a dense array to dictionary given known dictionary

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-3120 [C++] Parallelize execution of ScalarAggregateFunction

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-3978 [C++] Implement hashing, dictionary-encoding for StructArray

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-4097 [C++] Add function to "conform" a dictionary array to a target new dictionary

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-11090 [C++] Support temporal arithmetic ({time,date}{32,64}, timestamp, interval)

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1569 [C++] Kernel functions for determining monotonicity (ascending or descending) for well-ordered types

          • Major - Major loss of function.
          • In Progress

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1567 [C++] Implement "fill null" kernels that replace null values with some scalar replacement value

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1568 [C++] Implement "drop null" kernels that return array without nulls

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1699 [C++] Forward, backward fill kernel functions

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1846 [C++] Implement "any" reduction kernel for boolean data

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-1888 [C++] Implement casts from one struct type to another (with same field names and number of fields)

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-6978 [R] Add bindings for sum and mean compute kernels

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-7010 [C++] Support lossy casts from decimal128 to float32 and float64/double

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-7011 [C++] Implement casts from float/double to decimal128

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8922 [C++] Implement example string scalar kernel function to assist with string kernels buildout per ARROW-555

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8934 [C++] Add timestamp subtract kernel aliased to int64 subtract implementation

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8937 [C++] Add "parse_strptime" function for string to timestamp conversions using the kernels framework

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8938 [R] Provide binding for arrow::compute::CallFunction

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-13174 [C++][Compute] Add strftime kernel

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-7009 [C++] Refactor filter/take kernels to use Datum instead of overloads

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-3122 [C++] Incremental Variance, Standard Deviation aggregators

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-3802 [C++] Cast to/from halffloat not implemented

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-5005 [C++] Implement support for using selection vectors in scalar aggregate function kernels

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-5530 [C++] Add options to ValueCount/Unique/DictEncode kernel to toggle null behavior

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-5890 [C++][Python] Support ExtensionType arrays in more kernels

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-7245 [C++] Allow automatic String -> LargeString promotions when concatenating tables

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8897 [C++] Determine strategy for propagating failures in initializing built-in function registry in arrow/compute

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8898 [C++] Determine desirable maximum length for ExecBatch in pipelined and parallel execution of kernels

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8921 [C++] Add "TypeResolver" class interface to replace current OutputType::Resolver pattern

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8936 [C++] Parallelize execution of arrow::compute::ScalarFunction

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9003 [C++] Add VectorFunction wrapping arrow::Concatenate

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9006 [C++] Deprecate or remove Scalar::Parse and Scalar::CastTo

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-12748 [C++] Arithmetic kernels for numeric arrays

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13121 [C++][Compute] Extract preallocation logic from KernelExecutor

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13122 [C++][Compute] Dispatch* should examine options as well as input types

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13339 [C++] Implement hash_aggregate kernels (umbrella issue)

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-971 [C++/Python] Implement Array.isvalid/notnull/isnull as scalar functions

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-2665 [Python/C++] Add index() method to find first occurence of Python scalar

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-5854 [Python] Expose compare kernels on Array class

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6456 [C++] Possible to reduce object code generated in compute/kernels/take.cc?

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6982 [R] Add bindings for compare and boolean kernels

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-7017 [C++] Refactor AddKernel to support other operations and types

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-7179 [C++][Compute] Consolidate fill_null and coalesce

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8025 [C++] Implement cast to Binary and FixedSizeBinary

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8500 [C++] Use selection vectors in Filter implementation for record batches, tables

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8876 [C++] Implement casts from date types to Timestamp

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8895 [C++] Add C++ unit tests for filter and take functions on temporal type inputs, including timestamps

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8896 [C++] Reimplement dictionary unpacking in Cast kernels using Take

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8901 [C++] Reduce number of take kernels

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8903 [C++] Implement optimized "unsafe take" for use with selection vectors for kernel execution

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8917 [C++][Compute] Formalize "metafunction" concept

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8918 [C++] Add cast "metafunction" to FunctionRegistry that addresses dispatching to appropriate type-specific CastFunction

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8919 [C++] Add "DispatchBest" APIs to compute::Function that selects a kernel that may require implicit casts to invoke

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8923 [C++] Improve usability of arrow::compute::CallFunction by moving ExecContext* argument to end and adding default

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8926 [C++] Improve docstrings in new public APIs in arrow/compute and fix miscellaneous typos

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8928 [C++] Measure microperformance associated with ExecBatchIterator

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8929 [C++] Change compute::Arity:VarArgs min_args default to 0

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8933 [C++] Reduce generated code in vector_hash.cc

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8969 [C++] Reduce generated code in compute/kernels/scalar_compare.cc

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8976 [C++] compute::CallFunction can't Filter/Take with ChunkedArray

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9029 [C++] Implement BitBlockCounter interface for blockwise popcounts of validity bitmaps

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9045 [C++] Improve and expand Take/Filter benchmarks

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9055 [C++] Add sum/mean kernels for Boolean type

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9056 [C++] Support scalar aggregation over scalars

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-11928 [C++][Compute] Add ExecNode hierarchy

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-11929 [C++][Compute] Promote Expression to the compute namespace

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-11930 [C++][Dataset][Compute] Refactor Dataset scans to use an ExecNode graph

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-12499 [C++][Compute][R] Add ScalarAggregateOptions to Any and All kernels

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-12980 [C++] Kernels to extract datetime components should be timezone aware

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13005 [C++] Support filter/take for union data type.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13025 [C++][Compute] Enhance FunctionOptions with equality, debug representability, and serializability

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13033 [C++] Kernel to localize naive timestamps to a timezone (preserving clock-time)

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13054 [C++] Add option to specify the first day of the week for the "day_of_week" temporal kernel

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13064 [C++] Add a general "if, ifelse, ..., else" kernel ("CASE WHEN")

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13220 [C++] Add a 'choose' kernel/scalar compute function

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13222 [C++] Support variable-width types in case_when function

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13548 [C++] Implement datediff kernel

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-13549 [C++] Implement timestamp to date/time cast that extracts value

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6071 [C++] Implement casting Binary <-> LargeBinary

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6072 [C++] Implement casting List <-> LargeList

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6122 [C++] SortToIndices kernel must support FixedSizeBinary

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6123 [C++] ArgSort kernel should not materialize the output internal

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6923 [C++] Option for Filter kernel how to handle nulls in the selection vector

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-7083 [C++] Determine the feasibility and build a prototype to replace compute/kernels with gandiva kernels

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8916 [Python] Add relevant glue for implementing each kind of FunctionOptions

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-12053 [C++] Implement aggregate compute functions for decimal datatypes

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6974 [C++] Refactor temporal casts to work with Scalar inputs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Sub-task - The sub-task of the issue ARROW-9042 [C++] Add Subtract and Multiply arithmetic kernels with wrap-around behavior

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8939 [C++] Arrow-native C++ Data Frame-style programming interface for analytics (umbrella issue)

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. ARROW-3520 [C++] Implement List Flatten kernel

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-4333 [C++] Sketch out design for kernels and "query" execution in compute layer

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-5760 [C++] Optimize Take implementation

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. ARROW-8961 [C++] Add utf8proc library to toolchain

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8891 [C++] Split non-cast compute kernels into a separate shared library

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-7871 [Python] Expose more compute kernels

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8966 [C++] Move arrow::ArrayData to a separate header file

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8989 [C++] Document available functions in compute::FunctionRegistry

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-9075 [C++] Optimize Filter implementation

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8772 [C++] Expand SumKernel benchmark to more types

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-6990 [C++] Support casting between decimal types with compatible precision/scales

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8214 [C++] Flatbuffers based serialization protocol for Expressions

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8905 [C++] Collapse Take APIs from 8 to 1 or 2

          • Major - Major loss of function.
          • Closed

          Wish - General wishlist item. ARROW-6775 [C++] [Python] Proposal for several Array utility functions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. ARROW-8792 [C++] Improved declarative compute function / kernel development framework, normalize calling conventions

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              wesm Wes McKinney
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: