[IMPALA-10419] pytest hits UnicodeDecodeError in reporting assert failures for UTF-8 results

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Infrastructure
• Epic Link: UTF-8
• Epic Color: ghx-label-13
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 04/Jan/21 02:59
• Updated: 06/Jan/21 04:51
• Resolved: 06/Jan/21 04:51

Description

When the result contains non-ascii bytes, pytest fails to report assertion failures:

tests/query_test/test_exprs.py:71: in test_special_strings
    self.run_test_case('QueryTest/special-strings', vector)
tests/common/impala_test_suite.py:693: in run_test_case
    self.__verify_results_and_errors(vector, test_section, result, use_db)
tests/common/impala_test_suite.py:529: in __verify_results_and_errors
    replace_filenames_with_placeholder)
tests/common/test_result_verifier.py:465: in verify_raw_results
    VERIFIER_MAP[verifier](expected, actual)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/reinterpret.py:46: in __init__
    self.msg = reinterpret(source, f, should_fail=True)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/reinterpret.py:96: in reinterpret
    visitor.visit(mod)
toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/ast.py:241: in visit
    return visitor(node)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/reinterpret.py:191: in visit_Module
    self.visit(stmt)
toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/ast.py:241: in visit
    return visitor(node)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/reinterpret.py:383: in visit_Assert
    test_explanation, test_result = self.visit(assrt.test)
toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/ast.py:241: in visit
    return visitor(node)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/reinterpret.py:232: in visit_Compare
    res = util._reprcompare(op_symbol, left_result, next_result)
infra/python/env-gcc7.5.0/lib/python2.7/site-packages/_pytest/assertion/__init__.py:132: in callbinrepr
    res = py.builtin._totext("\n~").join(new_expl)
E   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

I can manually fix this by modifying infra/python/env-gcc7.5.0/lib/python2.7/site-packages/pytest/assertion/init_.py:131 by explicitly specifiying 'utf-8' as the decode encoding.

-                 new_expl = [line.replace("\n", "\\n") for line in new_expl]
+                 new_expl = [line.replace("\n", "\\n").decode('utf-8') for line in new_expl]
                 res = py.builtin._totext("\n~").join(new_expl)

Some versions of my env: Python 2.7.16, pytest-2.9.2, py-1.4.32, pluggy-0.3.1. We may need to bump pytest version for a fix.