[SPARK-35077] Migrate to transformWithPruning for leftover optimizer rules

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: Optimizer
• Assignee: Yingyi Bu
• Reporter: Yingyi Bu
• Created: 14/Apr/21 19:54
• Updated: 02/Jun/21 03:47
• Resolved: 02/Jun/21 03:46

Description

E.g., PushDownPredicates and a few others.

 

Commit https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631 contains the framework level change and a few example rule changes.

 

Example patterns:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala#L24-L32

 

Example rule:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala

 

https://github.com/apache/spark/pull/32247 is another example

Issue Links

• links to: [Github] Pull Request #32721 (sigmod)