[SPARK-44219] Add extra per-rule validation for optimization rewrites.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.4.1
• Fix Version/s: 4.0.0
• Component/s: Optimizer
• Labels: pull-request-available
• Assignee: Yannis Sismanis
• Reporter: Yannis Sismanis
• Shepherd: Wenchen Fan
• Created: 27/Jun/23 21:47
• Updated: 06/Oct/23 03:20
• Resolved: 06/Oct/23 03:20

Description

Adds per-rule validation checks for the following:

1.  aggregate expressions in Aggregate plans are valid.
2. Grouping key types in Aggregate plans cannot by of type Map. 
3. No dangling references have been generated.

This is validation is by default enabled for all tests or selectively using the spark.sql.planChangeValidation=true flag.

Issue Links

• links to: [Github] Pull Request #41763 (YannisSismanis)
• links to: GitHub Pull Request #41763