[ARROW-10547] [Rust][DataFusion] Filter pushdown loses filters if below a user defined node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Rust - DataFusion
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26513
• Assignee: Andrew Lamb
• Reporter: Andrew Lamb
• Created: 10/Nov/20 16:22
• Updated: 11/Jan/23 08:13
• Resolved: 11/Nov/20 06:36

Description

I have a LogicalPlan like this (where "Extension" is some extension node type):

    Extension [non_null_column:Utf8]
      Filter: #state Eq Utf8("MA") [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
        InMemoryScan: projection=None [city:Utf8;N, state:Utf8;N, borough:Utf8;N]

When I run this plan through ExecutionContext::optimize the plan that results is as follows (note the filter has been lost):

    Extension [non_null_column:Utf8]
      InMemoryScan: projection=Some([0, 1, 2]) [city:Utf8;N, state:Utf8;N, borough:Utf8;N]

I have debugged the problem and the root cause of the issue is that the FilterPushdown logic is not recursing into the inputs of user defined nodes. I will get a PR up showing and fixing the problem.

More generally, I am not sure how / if the filter pushdown logic would work if there are multiple child inputs as no existing LogicalPlan variant his multiple inputs at this time. We will have to handle this when / if we want to support joins (typically filters get pushed down each join input if possible)

Issue Links

• links to: GitHub Pull Request #8633