[SPARK-32276] Remove redundant sorts before repartition nodes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Anton Okolnychyi
• Reporter: Anton Okolnychyi
• Created: 10/Jul/20 22:34
• Updated: 19/Jul/20 19:12
• Resolved: 19/Jul/20 19:12

Description

I think our EliminateSorts rule can be extended further to remove sorts before repartition, repartitionByExpression and coalesce nodes. Independently of whether we do a shuffle or not, each repartition operation will change the ordering and distribution of data.

That's why we should be able to rewrite Repartition -> Sort -> Scan as Repartition -> Scan.

Issue Links

• is related to: SPARK-32318 Add a test case to EliminateSortsSuite for protecting ORDER BY in DISTRIBUTE BY
• links to: [Github] Pull Request #29089 (aokolnychyi)
• links to: [Github] Pull Request #29089 (aokolnychyi)