[SPARK-33612] Add dataSourceRewriteRules batch to Optimizer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Anton Okolnychyi
• Reporter: Anton Okolnychyi
• Created: 30/Nov/20 17:59
• Updated: 15/Dec/20 02:00
• Resolved: 01/Dec/20 17:28

Description

Right now, we have a special place in the optimizer where we run rules for constructing v2 scans. As time shows, we need more rewrite rules for v2 tables and not all of them related to reads. One option is to rename the current batch into something more generic but it would require changing quite some places. Moreover, the batch contains some non-V2 logic as well. That's why it seems better to introduce a new batch and use it for all data source v2 rewrites and beyond.

Issue Links

• is related to: SPARK-33784 Rename dataSourceRewriteRules and customDataSourceRewriteRules in BaseSessionStateBuilder
• relates to: SPARK-33621 Add a way to inject data source rewrite rules
• links to: [Github] Pull Request #30558 (aokolnychyi)
• links to: [Github] Pull Request #30558 (aokolnychyi)