[IMPALA-8130] VERIFY_IF_SUBSET gives false positives on dev box

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 3.1.0
• Component/s: Infrastructure
• Epic Color: ghx-label-5
• Assignee: Unassigned
• Reporter: Paul Rogers
• Created: 27/Jan/19 21:23
• Updated: 27/Jan/19 21:23

Description

Introduced a change that caused a (yet unknown) change in the behavior for tables partitioned using a column of type double. Ran the pre-review tests and got a failure:

00:50:52  TestPartitioning.test_partition_col_types[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
00:50:52 [gw2] linux2 -- Python 2.7.12 /home/ubuntu/Impala/bin/../infra/python/env/bin/python
00:50:52 query_test/test_partitioning.py:45: in test_partition_col_types
00:50:52     use_db=unique_database)
00:50:52 common/impala_test_suite.py:501: in run_test_case
00:50:52     self.__verify_results_and_errors(vector, test_section, result, use_db)
00:50:52 common/impala_test_suite.py:359: in __verify_results_and_errors
00:50:52     replace_filenames_with_placeholder)
00:50:52 common/test_result_verifier.py:449: in verify_raw_results
00:50:52     VERIFIER_MAP[verifier](expected, actual)
00:50:52 common/test_result_verifier.py:239: in verify_query_result_is_subset
00:50:52     assert expected_literal_strings <= actual_literal_strings
00:50:52 E   assert Items in expected results not found in actual results:
00:50:52 E     '   partitions=10/11 files=10 size=20B'

Notice the last line: we expect to match 10 partitions. This output does not show the actual output, nor is the actual output saved in any of the logs that I could find. (This is another nuisance that should be fixed...)

Ran the same test on the same code on my dev box. This time the test passes:

[gw3] PASSED query_test/test_partitioning.py::TestPartitioning::test_partition_col_types[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] -- 2019-01-27 11:53:16,531 INFO     MainThread: Started query df42090017b0b652:4647c82500000000

However, if I examine the expected and actual output, in file QueryTest/partition-col-types.test, I see that the test actually did fail locally, but was not reported as a failure. Expected results:

---- QUERY
EXPLAIN
SELECT * from all_insert_partition_col_types
WHERE double_col = 1.1
---- RESULTS: VERIFY_IS_SUBSET
'01:EXCHANGE [UNPARTITIONED]'
'|'
'00:SCAN HDFS [$DATABASE.all_insert_partition_col_types]'
'   partitions=10/11 files=10 size=20B'

Notice the last line: it is the one that was expected, but not found, in the pre-review tests. Actual output:

---- RESULTS: VERIFY_IS_SUBSET
...
'PLAN-ROOT SINK'
'|'
'01:EXCHANGE [UNPARTITIONED]'
'|'
'00:SCAN HDFS [test_partition_col_types_a253a8a3.all_insert_partition_col_types]'
'   partition predicates: double_col = 1.1'
'   partitions=0/11 files=0 size=0B'
'   row-size=42B cardinality=0'

Notice the second-to-last line. It says that we matched zero partitions, which is a bug because we expect to match 10 as per the earlier lines.

However, despite the error, and that the expected and actual files should not match, the test run was reported as a success.

To reproduce this, grab patch set 3 from the following patch:

12265 (Draft): IMPALA-8106: Cleanup of literal expression nodes

This is currently a draft; will be published once the bug has been found and fixed.