[FLINK-24717] Push down partitions before filters

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Table SQL / Planner
• Labels: pull-request-available

Release Note

This might have an impact on existing table source implementations as push down filters might not contain partition predicates anymore. However, the connector implementation for table sources that implement both partition and filter push down became easier with this change.

• Assignee: xuyang
• Reporter: Timo Walther
• Created: 01/Nov/21 11:43
• Updated: 08/Nov/21 14:39
• Resolved: 08/Nov/21 14:39

Description

Currently, we push filters before partitions. This means that a applyFilters needs to have partition logic to extract the partition predicate. Furthermore, if a applyFilters consumes all filters (no remaining predicates), the applyPartitions is never called.

We should execute the PushPartitionIntoTableSourceScanRule first and check for side effects of this change.

See org.apache.flink.table.planner.plan.rules.logical.PushProjectIntoTableSourceScanRuleTest#testMetadataProjectionWithoutProjectionPushDownWhenNotSupportedAndNoneSelected for an example of using the new test infrastructure.

Issue Links

• links to: GitHub Pull Request #17652