[SPARK-44505] DataSource v2 Scans should not require planning the input partitions on explain

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: SQL
• Assignee: Martin Grund
• Reporter: Martin Grund
• Created: 20/Jul/23 21:55
• Updated: 27/Jul/23 12:54
• Resolved: 27/Jul/23 12:54

Description

Right now, we will always call `planInputPartitions()` for a DSv2 implementation even if there is no spark job run but only explain.

We should provide a way to avoid scanning all input partitions just to determine if the input is columnar or not. The scan should provide an override.

Issue Links

• links to: [Github] Pull Request #42099 (grundprinzip)