[SPARK-32466] Add support to catch SparkPlan regression base on TPC-DS queries

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: wuyi
• Reporter: wuyi
• Created: 28/Jul/20 11:41
• Updated: 26/Aug/20 02:16
• Resolved: 17/Aug/20 14:22

Description

Nowadays, Spark is getting more and more complex. Any changes might cause regression unintentionally. Spark already has some benchmark to catch the performance regression. But, yet, it doesn't have a way to detect the regression inside SparkPlan. It would be good if we could find some possible regression early during the compile phase before the runtime phase.

Issue Links

• fixes: SPARK-25872 Add an optimizer tracker for TPC-DS queries
• links to: [Github] Pull Request #29270 (Ngone51)
• links to: [Github] Pull Request #29537 (Ngone51)
• links to: [Github] Pull Request #29546 (Ngone51)