[ARROW-16164] [C++] Pushdown filters on augmented columns like fragment filename

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/20187
• Assignee: Unassigned
• Reporter: Nicola Crane
• Created: 11/Apr/22 15:11
• Updated: 11/Jan/23 11:42

Description

In the discussion on ARROW-15260, if we run the following code in R, we might expect it to push down the filter so we can just read in the relevant files:

  filter = Expression$create(
    "match_substring",
    Expression$field_ref("__filename"),
    options = list(pattern = "cyl=8")
  )

As mentioned by westonpace:

"You might think we would get the hint and only read files matching that pattern. This is not the case. We will read the entire dataset and apply the "cyl=8" filter in memory.

If we want to pushdown filters on the filename column we will need to add some special logic."

Issue Links

• is related to: ARROW-15260 [R] open_dataset - add file_name as column