[SPARK-17059] Allow FileFormat to specify partition pruning strategy

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Andrew Duffy
• Created: 15/Aug/16 13:03
• Updated: 21/May/19 04:32
• Resolved: 21/May/19 04:32

Description

Allow Spark to have pluggable pruning of input files for FileSourceScanExec by allowing FileFormat's to specify format-specific filterPartitions method.

This is especially useful for Parquet as Spark does not currently make use of the summary metadata, instead reading the footer of all part files for a Parquet data source. This can lead to massive speedups when reading a filtered chunk of a dataset, especially when using remote storage (S3).

Issue Links

• links to: [Github] Pull Request #14649 (andreweduffy)
• links to: [Github] Pull Request #15835 (pwoody)