[HIVE-23965] Improve plan regression tests using TPCDS30TB metastore dump and custom configs

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 31/Jul/20 15:17
• Updated: 27/Feb/24 22:23
• Resolved: 04/Dec/20 13:32

Description

The existing regression tests (HIVE-12586) based on TPC-DS have certain shortcomings:

The table statistics do not reflect cardinalities from a specific TPC-DS scale factor (SF). Some tables are from a 30TB dataset, others from 200GB dataset, and others from a 3GB dataset. This mix leads to plans that may never appear when using an actual TPC-DS dataset.

The existing statistics do not contain information about partitions something that can have a big impact on the resulting plans.

The existing regression tests rely on more or less on the default configuration (hive-site.xml). In real-life scenarios though some of the configurations differ and may impact the choices of the optimizer.

This issue aims to address the above shortcomings by using a curated TPCDS30TB metastore dump along with some custom hive configurations.

Issue Links

• fixes: HIVE-12587 Support to add partitioned data set to TestPerfCliDriver
• is related to: HIVE-27216 Upgrade postgresql to 42.5.1 from 9.x
• is related to: HIVE-26830 Update TPCDS30TB metastore dump with histograms
• is related to: HIVE-24439 HS2 memory leak when commitTxn fails and queries involve partitioned tables
• is related to: HIVE-24480 Remove metastore upgrade script from 3.1.3000 to 4.0.0
• relates to: HIVE-27382 RDMBS docker images: use official images or contribute Dockerfile to hive repo
• relates to: HIVE-12586 Support for Performance regression test suite
• links to: GitHub Pull Request #1347
• links to: GitHub Pull Request #1714