[SPARK-46693] Inject LocalLimitExec when matching OffsetAndLimit or LimitAndOffset

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Optimizer
• Labels: pull-request-available
• Assignee: Nick Young
• Reporter: Nick Young
• Created: 12/Jan/24 04:14
• Updated: 16/Jan/24 03:02
• Resolved: 13/Jan/24 21:18

Description

For queries containing both a LIMIT and an OFFSET in a subquery, physical translation will drop the `LocalLimit` planned in the optimizer stage by mistake; this manifests as larger than necessary shuffle sizes for `GlobalLimitExec`. Fix to not drop this node.

Issue Links

• links to: GitHub Pull Request #44699