[SPARK-35144] Migrate to transformWithPruning or resolveWithPruning for object rules

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Optimizer
• Assignee: Yingyi Bu Assign to me
• Reporter: Yingyi Bu
• Created: 20/Apr/21 07:27
• Updated: 07/May/21 10:36
• Resolved: 07/May/21 10:36

Description

Rules in org/apache/spark/sql/catalyst/optimizer/objects.Scala

 

Commit https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631 contains the framework level change and a few example rule changes.

 

Example patterns:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala#L24-L32

 

Example rule:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala

 

https://github.com/apache/spark/pull/32247 is another example

Issue Links

• links to: [Github] Pull Request #32451 (sigmod)