[SPARK-35147] Migrate to resolveWithPruning for two command rules

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.1.0
• Component/s: Optimizer
• Assignee: Unassigned
• Reporter: Yingyi Bu
• Created: 20/Apr/21 08:01
• Updated: 03/Jun/21 02:27
• Resolved: 03/Jun/21 02:27

Description

We can add one TreePattern called "COMMAND". Then, two rules to be migrated:

• ResolvePartitionSpec
• ResolveCommandsWithIfExists 

 

Commit https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631 contains the framework level change and a few example rule changes.

 

Example patterns:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala#L24-L32

 

Example rule:

https://github.com/apache/spark/blob/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala

 

https://github.com/apache/spark/pull/32247 is another example

Issue Links

• links to: [Github] Pull Request #32463 (sigmod)
• links to: [Github] Pull Request #32463 (sigmod)