[SPARK-32096] Improve sorting performance for Spark SQL rank window function​

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL

Environment

Any environment that supports Spark.

• Assignee: Unassigned
• Reporter: Zikun
• Created: 24/Jun/20 21:12
• Updated: 21/Sep/20 05:00

Description

Spark SQL rank window function needs to sort the data in each window partition, and it relies on the execution operator  SortExec to do the sort. During sorting, the window partition key is also put at the front of the sort order and thus it brings unnecessary comparisons on the partition key. Instead, we can group the rows by partition key first, and inside each group we sort the rows without comparing the partition key.​ 

 

The Jira https://issues.apache.org/jira/browse/SPARK-32947 is a follow-up effort of this improvement.

Issue Links

• links to: [Github] Pull Request #29725 (xuzikun2003)
• links to: [Github] Pull Request #29725 (xuzikun2003)