[IMPALA-8315] Error from shutil.rmtree in ImpalaTestSuite::run_stmt_in_hive()

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: broken-build
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Joe McDonnell
• Created: 15/Mar/19 22:07
• Updated: 26/Jun/19 20:16
• Resolved: 26/Jun/19 20:16

Description

TestHBaseQueries.test_hbase_col_filter() failed on a centos6 exhaustive build with the following error:

query_test/test_hbase_queries.py:89: in test_hbase_col_filter
    self.run_stmt_in_hive(add_data)
common/impala_test_suite.py:781: in run_stmt_in_hive
    if tmpdir is not None: shutil.rmtree(tmpdir)
/usr/lib64/python2.6/shutil.py:212: in rmtree
    rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:212: in rmtree
    rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:212: in rmtree
    rmtree(fullname, ignore_errors, onerror)
/usr/lib64/python2.6/shutil.py:217: in rmtree
    onerror(os.remove, fullname, sys.exc_info())
/usr/lib64/python2.6/shutil.py:215: in rmtree
    os.remove(fullname)
E   OSError: [Errno 2] No such file or directory: '/tmp/impala-tests-j2b9bQ/localRunner/jenkins/job_local541814598_0023/job_local541814598_0023.xml'

ImpalaTestSuite::run_stmt_in_hive() creates a temporary directory when in LocalRunner mode to avoid different tests conflicting on the directory name. It creates the temp directory, runs the Hive statement, then removes the directory at the end. The removal is failing here. It looks like there might be a race between pytest running shutil.rmtree() and Hive cleaning up its own files from the statement.

We might want to use shutil.rmtree(..., ignore_errors=True).

Seen once.