Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10322

[C++][Dataset] Minimize Expression to a wrapper around compute::Function

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.1
    • Fix Version/s: 3.0.0
    • Component/s: C++
    • Labels:
      None

      Description

      The Expression class hierarchy was originally intended to provide generic, structured representations of compute functionality. On the former point they have been superseded by compute::

      {Function, Kernel, ...}

      which encapsulates validation and execution. In light of this Expression can be drastically simplified and improved by composition with these classes. Each responsibility which can be deferred implies less boilerplate when exposing a new compute function for use in datasets. Ideally any compute function will be immediately available to use in a filter or projection.

      struct Expression {
        using Literal = std::shared_ptr<Scalar>;
      
        struct Projection {
          std::vector<std::string> names
          std::vector<Expression> values;
        };
      
        struct Call {
          std::shared_ptr<ScalarFunction> function;
          std::shared_ptr<FunctionOptions> options;
          std::vector<Expression> arguments;
        };
      
        util::variant<Literal, FieldRef, Projection, Call> value;
      };
      

      A simple discriminated union as above should be sufficient to represent arbitrary filters and projections: any expression which results in type bool is a valid filter, and any expression which is a Projection may be used to map one record batch to another.

      Expression simplification (currently implemented in Expression::Assume) is an optimization used for example in predicate pushdown, and therefore need not exhaustively cover the full space of available compute functions.

        Attachments

          Activity

            People

            • Assignee:
              bkietz Ben Kietzman
              Reporter:
              bkietz Ben Kietzman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: