[SPARK-33183] Bug in optimizer rule EliminateSorts

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 3.0.2, 3.1.0
• Fix Version/s: 2.4.8, 3.0.2, 3.1.0
• Component/s: SQL
• Labels: correctness
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 19/Oct/20 17:54
• Updated: 19/Nov/20 05:07
• Resolved: 28/Oct/20 05:52

Description

Currently, the rule EliminateSorts removes a global sort node if its child plan already satisfies the required sort order without checking if the child plan's ordering is local or global. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. 

Sort(orders, global = True, ...)
  Sort(orders, global = False, ...)

 

Issue Links

• is caused by: SPARK-23973 Remove consecutive sorts
• links to: [Github] Pull Request #30093 (allisonwang-db)
• links to: [Github] Pull Request #30194 (allisonwang-db)
• links to: [Github] Pull Request #30195 (allisonwang-db)
• links to: [Github] Pull Request #30373 (allisonwang-db)
• links to: [Github] Pull Request #30373 (allisonwang-db)
• links to: [Github] Pull Request #30420 (allisonwang-db)