[SPARK-38162] Optimize one row plan in normal and AQE Optimizer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: XiDuo You
• Reporter: XiDuo You
• Created: 09/Feb/22 10:18
• Updated: 13/Jun/23 23:45
• Resolved: 23/Feb/22 07:34

Description

Optimize the plan if its max row is equal to or less than 1 in these cases:

• if the child of sort max rows less than or equal to 1, remove the sort
• if the child of local sort max rows per partition less than or equal to 1, remove the local sort
• if the child of aggregate max rows less than or equal to 1 and it's grouping only (include the rewritten distinct plan), remove the aggregate
• if the child of aggregate max rows less than or equal to 1, set distinct to false in all aggregate expression

Issue Links

• is related to: SPARK-44040 Incorrect result after count distinct
• links to: [Github] Pull Request #35473 (ulysses-you)