[SPARK-27384] File source V2: Prune unnecessary partition columns

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 04/Apr/19 06:18
• Updated: 29/Apr/19 05:37
• Resolved: 08/Apr/19 07:16

Description

When scanning file sources, we can prune unnecessary partition columns on constructing input partitions, so that:
1. Reduce the data transformation from Driver to Executors
2. Make it easier to implement columnar batch readers, since the partition columns are already pruned.

Issue Links

• links to: GitHub Pull Request #24296