[IMPALA-13125] Set of tests for exploration_strategy=exhaustive varies between python 2 and 3

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 4.5.0
• Component/s: Infrastructure
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Joe McDonnell
• Created: 03/Jun/24 06:17
• Updated: 03/Jun/24 06:17

Description

TLDR: Python 3 runs a different set of exhaustive tests than Python 2.

Longer version:

When looking into running Python 3 tests, I noticed that the set of tests running for the exhaustive tests is different for Python 2 vs Python 3. This was surprising.

It turns out there is a distinction between run-tests.py's --exploration_strategy=exhaustive vs the --workload_exploration_strategy="functional-query:exhaustive" option. The exhaustive job is actually doing the latter. This means that individual function-query workload classes see cls.exploration_strategy() == "exhaustive", but the logic that generates the test vector still see exploration_strategy=core and it still uses pairwise generation. Code:

    if exploration_strategy == 'exhaustive':
      return self.__generate_exhaustive_combinations()
    elif exploration_strategy in ['core', 'pairwise']:
      return self.__generate_pairwise_combinations()

https://github.com/apache/impala/blob/master/tests/common/test_vector.py#L165-L168

Python 2 vs 3 changes the way dictionaries work, impacting the order of test dimensions and how it picks tests. So, the Python 3 exhaustive tests are different. This may expose latent bugs, because some combinations that meet the constraints are never actually run (e.g. some json encodings don't have the decimal_tiny table).

We can work to make them behave similarly, using pytest's --collect-only option to look at the differences (and compare them to actual existing runs).