[ARROW-11174] [C++][Dataset] Make Expressions available for projection

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27080
• Assignee: Ben Kietzman
• Reporter: Ben Kietzman
• Created: 07/Jan/21 21:56
• Updated: 11/Jan/23 08:18
• Resolved: 26/Feb/21 22:04

Description

RecordBatchProjector should be replaced by an expression calling the "project" compute function.

Projection currently supports only reordering and subselection of fields, materializing virtual columns where necessary. Replacement with an Expression will enable specifying arbitrary expressions for projected columns:

// project an explicit selection:
// SELECT a as "a", b as "b" ...
project({field_ref("a"), field_ref("b")}, {"a", "b"});

// project an arithmetic expression:
// SELECT a + b as "a + b" ...
project({add(field_ref("a"), field_ref("b"))}, {"a + b"})

This will also allow the same expression optimization machinery used for filters to be directly applied to projections. Virtual columns become a consequence of constant folding:

// project in a partition where a == 3:
assert(
  SimplifyWithGuarantee(
    project({field_ref("a"), field_ref("b")}, {"a", "b"}),
    equal(field_ref("a"), literal(3))
  )
  == project({literal(3), field_ref("b")}, {"a", "b"})
)

 

Issue Links

• fixes: ARROW-8630 [C++][Dataset] Pass schema including all materialized fields to catch CSV edge cases
• is depended upon by: ARROW-11704 [R] Wire up dplyr::mutate() for datasets
• links to: GitHub Pull Request #9532