[SPARK-33122] Remove redundant aggregates in the Optimzier

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Tanel Kiis
• Reporter: Tanel Kiis
• Created: 12/Oct/20 20:18
• Updated: 15/Jun/21 11:38
• Resolved: 20/Mar/21 02:18

Description

It is possible to have two or more consecutive aggregates whose sole purpose is to keep only distinct values (for example TPCDS q87). We can remove all but the last one do improve performance.

Issue Links

• is blocked by: SPARK-33225 Extract AliasHelper trait
• relates to: SPARK-35765 Distinct aggs are not duplicate sensitive
• links to: [Github] Pull Request #30018 (tanelk)
• links to: [Github] Pull Request #30018 (tanelk)
• links to: [Github] Pull Request #31914 (tanelk)
• links to: [Github] Pull Request #32904 (tanelk)