[SPARK-30475] File source V2: Push data filters for file listing

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Guy Khazma
• Reporter: Guy Khazma
• Created: 09/Jan/20 23:36
• Updated: 21/Jan/20 05:21
• Resolved: 21/Jan/20 04:23

Description

Follow up on SPARK-30428 which added support for partition pruning in File source V2.
We should also pass the dataFilters to the listFiles method.

Datasources such as csv and json do not implement the SupportsPushDownFilters trait. In order to support data skipping uniformly for all file based data sources, one can override the listFiles method in a FileIndex implementation and use the dataFilters and partitionFilters to consult external metadata and prunes the list of files.

Issue Links

• links to: GitHub Pull Request #27157