[IMPALA-10416] Testfile can't deal with non-ascii results

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Infrastructure
• Epic Link: UTF-8
• Epic Color: ghx-label-2
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 31/Dec/20 01:41
• Updated: 06/Jan/21 04:51
• Resolved: 06/Jan/21 04:51

Description

In the testfile, we can use non-ascii characters in the query. But when the result contains non-ascii characters, the test framework fails to deal with them.

For instance, I'm currently on master branch (commit 5baadd1). Add a simple test query:

diff --git a/testdata/workloads/functional-query/queries/QueryTest/special-strings.test b/testdata/workloads/functional-query/queries/QueryTest/special-strings.test
index 99a694c..9dfbc97 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/special-strings.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/special-strings.test
@@ -24,3 +24,10 @@ select "'"
 ---- TYPES
 STRING
 ====
+---- QUERY
+select "你好"
+---- RESULTS
+'你好'
+---- TYPES
+STRING
+==== 

Run the test

impala-py.test tests/query_test/test_exprs.py::TestExprs::test_special_strings 

The failure occurs:

TestExprs.test_special_strings[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none | enable_expr_rewrites: 1] 
tests/query_test/test_exprs.py:71: in test_special_strings
    self.run_test_case('QueryTest/special-strings', vector)
tests/common/impala_test_suite.py:693: in run_test_case
    self.__verify_results_and_errors(vector, test_section, result, use_db)
tests/common/impala_test_suite.py:529: in __verify_results_and_errors
    replace_filenames_with_placeholder)
tests/common/test_result_verifier.py:452: in verify_raw_results
    actual = QueryTestResult(parse_result_rows(exec_result), actual_types,
tests/common/test_result_verifier.py:493: in parse_result_rows
    col = cols[i].encode('unicode_escape')
E   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128) 

Issue Links

• blocks: IMPALA-2019 Proper UTF-8 support in string functions