[SPARK-27714] Support Join Reorder based on Genetic Algorithm when the # of joined tables > 12

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Xianyin Xin
• Reporter: Xianyin Xin
• Created: 15/May/19 02:36
• Updated: 05/Mar/23 09:04

Description

Now the join reorder logic is based on dynamic planning which can find the most optimized plan theoretically, but the searching cost grows rapidly with the # of joined tables grows. It would be better to introduce Genetic algorithm (GA) to overcome this problem.

Issue Links

• links to: GitHub Pull Request #24983
• links to: Introduction, Design and Experiment