[SPARK-32985] Decouple bucket filter pruning and bucket table scan

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Cheng Su
• Reporter: Cheng Su
• Created: 24/Sep/20 07:04
• Updated: 19/Aug/21 23:02
• Resolved: 05/Feb/21 13:00

Description

As a followup from discussion in https://github.com/apache/spark/pull/29804#discussion_r493100510 . Currently in data source v1 file scan `FileSourceScanExec`, bucket filter pruning will only take effect with bucket table scan - https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala#L542 . However this is unnecessary, as bucket filter pruning can also happen if we disable bucketed table scan. This help query leverage the benefit from bucket filter pruning to save CPU/IO to not read unnecessary bucket files, and do not bound by bucket table scan when the parallelism of tasks is a concern.

Issue Links

• fixes: SPARK-33207 Reduce the number of tasks launched after bucket pruning
• links to: [Github] Pull Request #31413 (c21)
• links to: [Github] Pull Request #32000 (c21)