[SPARK-48431] Do not forward predicates on collated columns to file readers

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Jan-Ole Sasse
• Reporter: Jan-Ole Sasse
• Created: 27/May/24 14:00
• Updated: 29/May/24 20:52
• Resolved: 29/May/24 20:52

Description

SPARK-47657 allows to push filters on collated columns to file sources that support it. If such filters are pushed to file sources, those file sources must not push those filters to the actual file readers (i.e. parquet or csv readers), because there is no guarantee that those support collations.

With this task, we are widening filters on collations to be AlwaysTrue when we translate filters for file sources.

Issue Links

• links to: GitHub Pull Request #46760