Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 2.7.0
-
None
Description
Background:
The random query generator proper used to test Impala has a very simple API. A caller instantiates a QueryGenerator object and repeatedly calls the object's create_query() method to get back a query. The returned query is a Query object, a python tree-like representation of our own design that has nothing to do with the syntactical differences of various SQLs (e.g., Impala and PostgresQL). Later, various SqlWriter objects are responsible for taking Query objects and translating them into syntactically valid SQL.
Implications:
The code path for actually generating a Query object is entirely within our control (modulo whatever Python modules we depend on). Exceptions raised within this path are our own bugs and do not represent bugs in Impala or a reference database. Exceptions raised here can get quashed during the regular full run of "the query generator" in the system sense, but we would prefer not to have to do that. It also means that if we want to test whether the QueryGenerator object is working, that's harder to tell if 1 out of 10 queries results in an exception.
Problem:
Unfortunately we're generating such exceptions.
Here's one:
Traceback (most recent call last):
File "tests/comparison/query_generator.py", line 1493, in <module>
query = query_generator.create_query(tables)
File "tests/comparison/query_generator.py", line 208, in create_query
from_clause.visible_table_exprs, basic_select_item_exprs)
File "tests/comparison/query_generator.py", line 1276, in _create_having_clause
predicate, _ = self._create_boolean_func_tree()
File "tests/comparison/query_generator.py", line 1425, in _create_boolean_func_tree
signature = self.profile.choose_relational_func_signature(relational_signatures)
File "/home/mikeb/Impala/tests/comparison/query_profile.py", line 445, in choose_relational_func_signature
% ", ".join([missing_funcs]))
TypeError: sequence item 0: expected string, set found
This is easy to fix. Here's the real error:
Traceback (most recent call last):
File "tests/comparison/query_generator.py", line 1493, in <module>
query = query_generator.create_query(tables)
File "tests/comparison/query_generator.py", line 174, in create_query
from_clause.visible_table_exprs, table_alias_prefix)
File "tests/comparison/query_generator.py", line 1262, in _create_where_clause
predicate, _ = self._create_boolean_func_tree(allow_subquery=True)
File "tests/comparison/query_generator.py", line 1425, in _create_boolean_func_tree
signature = self.profile.choose_relational_func_signature(relational_signatures)
File "/home/mikeb/Impala/tests/comparison/query_profile.py", line 444, in choose_relational_func_signature
raise Exception("Weights are missing for functions: {0}".format(missing_funcs))
Exception: Weights are missing for functions: set([<class 'funcs.And'>, <class 'funcs.If'>, <class 'funcs.Or'>, <class 'funcs.Coalesce'>])
There's also this:
Traceback (most recent call last):
File "tests/comparison/query_generator.py", line 1493, in <module>
query = query_generator.create_query(tables)
File "tests/comparison/query_generator.py", line 162, in create_query
table_exprs, table_alias_prefix, required_table_expr_col_type)
File "tests/comparison/query_generator.py", line 1091, in _create_from_clause
join_clause = self._create_join_clause(from_clause, table_exprs)
File "tests/comparison/query_generator.py", line 1163, in _create_join_clause
table_expr, join_table_expr_candidates)
File "tests/comparison/query_generator.py", line 1210, in _create_relational_join_condition
relational_col_types=table_exprs_by_col_types.keys())
File "tests/comparison/query_generator.py", line 1425, in _create_boolean_func_tree
signature = self.profile.choose_relational_func_signature(relational_signatures)
File "/home/mikeb/Impala/tests/comparison/query_profile.py", line 447, in choose_relational_func_signature
return self.choose_func_signature(signatures, self.weights('RELATIONAL_FUNCS'))
File "/home/mikeb/Impala/tests/comparison/query_profile.py", line 534, in choose_func_signature
idx = self._choose_from_weights(signature_weights)
File "/home/mikeb/Impala/tests/comparison/query_profile.py", line 238, in _choose_from_weights
numeric_choice = randint(1, total_weight)
File "/usr/lib/python2.7/random.py", line 240, in randint
return self.randrange(a, b+1)
File "/usr/lib/python2.7/random.py", line 216, in randrange
raise ValueError, "empty range for randrange() (%d,%d, %d)" % (istart, istop, width)
ValueError: empty range for randrange() (1,1, 0)
There may be more after these gets fixed......
Method of reproduction:
$ impala-python tests/comparison/query_generator.py > /dev/null