[SPARK-45929] support grouping set operation in dataframe api

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: JacobZheng
• Reporter: JacobZheng
• Created: 15/Nov/23 02:58
• Updated: 21/Nov/23 01:41
• Resolved: 21/Nov/23 01:41

Description

I am using spark dataframe api for complex calculations. When I need to use the grouping sets function, I can only convert the expression to sql via analyzedPlan and then splice these sql into a complex sql to execute. In some cases, this operation generates an extremely complex sql. executing this complex sql, antlr4 continues to consume a large amount of memory, similar to a memory leak scenario. If you can and rollup, cube function through the dataframe api to calculate these operations will be much simpler.

Issue Links

• links to: GitHub Pull Request #43813