[SPARK-41337] Add a physical rule to remove the partialLimitExec node

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jiaan Geng
• Created: 30/Nov/22 13:18
• Updated: 02/Dec/22 08:53
• Resolved: 02/Dec/22 08:53

Description

Currently, the planner WindowGroupLimit creates two WindowGroupLimitExec node.

We can add a physical rule to remove the partialLimitExec node, if there is no shuffle between the two nodes (partialLimitExec's outputPartitioning satisfies the finalLimitExec's requiredChildDistribution)