[SPARK-34922] Use better CBO cost function

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.0.3, 3.1.2, 3.2.0
• Component/s: SQL
• Assignee: Tanel Kiis
• Reporter: Tanel Kiis
• Created: 31/Mar/21 12:01
• Updated: 08/Apr/21 02:04
• Resolved: 07/Apr/21 02:33

Description

In SPARK-33935 we changed the CBO cost function such that it would be symetric - A.betterThan(B) implies that !B.betterThan(A). Before both could have been true.

That change introduced a performance regression in some queries.

Issue Links

• links to: [Github] Pull Request #32014 (tanelk)
• links to: [Github] Pull Request #32075 (tanelk)
• links to: [Github] Pull Request #32076 (tanelk)