[ARROW-10322] [C++][Dataset] Minimize Expression to a wrapper around compute::Function

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.1
• Fix Version/s: 3.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26312
• Assignee: Ben Kietzman
• Reporter: Ben Kietzman
• Created: 16/Oct/20 00:59
• Updated: 11/Jan/23 08:12
• Resolved: 06/Jan/21 22:32

Description

The Expression class hierarchy was originally intended to provide generic, structured representations of compute functionality. On the former point they have been superseded by compute::

{Function, Kernel, ...}

which encapsulates validation and execution. In light of this Expression can be drastically simplified and improved by composition with these classes. Each responsibility which can be deferred implies less boilerplate when exposing a new compute function for use in datasets. Ideally any compute function will be immediately available to use in a filter or projection.

struct Expression {
  using Literal = std::shared_ptr<Scalar>;

  struct Call {
    std::shared_ptr<ScalarFunction> function;
    std::shared_ptr<FunctionOptions> options;
    std::vector<Expression> arguments;
  };

  util::variant<Literal, FieldRef, Call> value;
};

A simple discriminated union as above should be sufficient to represent arbitrary filters and projections: any expression which results in type bool is a valid filter, and any expression which is a Projection may be used to map one record batch to another.

Expression simplification (currently implemented in Expression::Assume) is an optimization used for example in predicate pushdown, and therefore need not exhaustively cover the full space of available compute functions.

Issue Links

• fixes: ARROW-8876 [C++] Implement casts from date types to Timestamp
• fixes: ARROW-10097 [C++] Persist SetLookupState in between usages of IsIn when filtering dataset batches
• supercedes: ARROW-9945 [C++][Dataset] Refactor Expression::Assume to return a Result
• links to: GitHub Pull Request #8894