[SPARK-35146] Migrate to transformWithPruning or resolveWithPruning for rules in finishAnalysis

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Optimizer
• Assignee: Yingyi Bu
• Reporter: Yingyi Bu
• Created: 20/Apr/21 07:53
• Updated: 11/May/21 09:12
• Resolved: 11/May/21 09:12

Description

Rules in org/apache/spark/sql/catalyst/optimizer/finishAnalysis.Scala, except RewriteNonCorrelatedExists which was done in SPARK-35075.

 

Commit https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631 contains the framework level change and a few example rule changes.

 

Example patterns:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala#L24-L32

 

Example rule:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala

 

https://github.com/apache/spark/pull/32247 is another example

Issue Links

• links to: [Github] Pull Request #32461 (sigmod)