[SPARK-30425] FileScan of Data Source V2 doesn't implement Partition Pruning

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Haifeng Chen
• Shepherd: Wenchen Fan
• Created: 06/Jan/20 03:06
• Updated: 08/Jan/20 07:35
• Resolved: 08/Jan/20 07:35

Description

I was trying to understand how Data Source V2 handling partition pruning,  I didn't find the code anywhere which filtering out the unnecessary files in current Data Source V2 implementation. For a File data source, the base class FileScan of Data Source V2 possibly should handle this in "partitions" method. But the current implementation is like the following:

protected def partitions: Seq[FilePartition] = {
val selectedPartitions = fileIndex.listFiles(Seq.empty, Seq.empty)

 

listFiles passed to empty sequence where no files will be filtered by the partition filter.

Issue Links

• duplicates: SPARK-30428 File source V2: support partition pruning