[IMPALA-9345] Hang in TestOrc::test_type_conversions when running create table in hive

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: Impala 3.4.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Epic Color: ghx-label-4
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 30/Jan/20 20:11
• Updated: 07/Feb/20 19:40
• Resolved: 07/Feb/20 19:40

Description

query_test.test_scanners.TestOrc.test_type_conversions[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: orc/def/block] (from pytest)
Failing for the past 1 build (Since Failed#93 )
Took 2 hr 0 min.
add description
Error Message

query_test/test_scanners.py:1337: in test_type_conversions     self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database) common/impala_test_suite.py:659: in run_test_case     result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:610: in __exec_in_hive     result = h.execute(query, user=user) common/impala_connection.py:334: in execute     r = self.__fetch_results(handle, profile_format=profile_format) common/impala_connection.py:441: in __fetch_results     cursor._wait_to_finish() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish     resp = self._last_operation._rpc('GetOperationStatus', req) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc     response = self._execute(func_name, request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute     return func(request) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus     return self.recv_GetOperationStatus() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus     (fname, mtype, rseqid) = iprot.readMessageBegin() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin     sz = self.readI32() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32     buff = self.trans.readAll(4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll     chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read     self._read_frame() /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame     header = read_all_compat(self._trans, 4) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>     read_all_compat = lambda trans, sz: trans.readAll(sz) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll     chunk = self.read(sz - have) /data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read     buff = self.handle.recv(sz) E   Failed: Timeout >7200s

Stacktrace

query_test/test_scanners.py:1337: in test_type_conversions
    self.run_test_case('DataErrorsTest/orc-type-checks', vector, unique_database)
common/impala_test_suite.py:659: in run_test_case
    result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
common/impala_test_suite.py:610: in __exec_in_hive
    result = h.execute(query, user=user)
common/impala_connection.py:334: in execute
    r = self.__fetch_results(handle, profile_format=profile_format)
common/impala_connection.py:441: in __fetch_results
    cursor._wait_to_finish()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:405: in _wait_to_finish
    resp = self._last_operation._rpc('GetOperationStatus', req)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:992: in _rpc
    response = self._execute(func_name, request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/hiveserver2.py:1009: in _execute
    return func(request)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:517: in GetOperationStatus
    return self.recv_GetOperationStatus()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py:529: in recv_GetOperationStatus
    (fname, mtype, rseqid) = iprot.readMessageBegin()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:126: in readMessageBegin
    sz = self.readI32()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py:206: in readI32
    buff = self.trans.readAll(4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
    chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:159: in read
    self._read_frame()
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/__init__.py:163: in _read_frame
    header = read_all_compat(self._trans, 4)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/infra/python/env/lib/python2.7/site-packages/thrift_sasl/six.py:31: in <lambda>
    read_all_compat = lambda trans, sz: trans.readAll(sz)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TTransport.py:58: in readAll
    chunk = self.read(sz - have)
/data/jenkins/workspace/impala-asf-master-core-s3-data-cache/Impala-Toolchain/thrift-0.9.3-p7/python/lib64/python2.7/site-packages/thrift/transport/TSocket.py:105: in read
    buff = self.handle.recv(sz)
E   Failed: Timeout >7200s

This has happened on multiple builds, all running against S3. It seems to be happening with high frequency on S3, but isn't deterministic - I saw this commit both succeed and fail:

    IMPALA-9334: Log node blacklisting removals in ClusterMembershipMgr

All failures did occur after " IMPALA-9242: Bump CDH_BUILD_NUMBER to 1814051" but not sure if that's related or not.

Sample hive log:

tarmstrong@tarmstrong-box:~/tmplogs/impala-asf-master-core-s3-data-cache-93/logs/cluster$ grep a6c81474e8c hive*
grep: hive: Is a directory
hive-server2.log:2020-01-28T11:10:49,558  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9): create external table test_type_conversions_3e81315e.union_complextypes(
hive-server2.log:2020-01-28T11:10:50,798  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] ql.Driver: Completed compiling command(queryId=jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9); Time taken: 1.274 seconds
hive-server2.log:2020-01-28T13:10:16,567  INFO [e00c5c82-1a05-4fc7-bed2-1c62751c2223 HiveServer2-Handler-Pool: Thread-41] operation.Operation: The running operation has been successfully interrupted: jenkins_20200128111049_91b0e648-7740-431f-88bb-a6c81474e8c9

Issue Links

• is broken by: IMPALA-9324 HdfsOrcScanner crashes in DCHECK failure in OrcSchemaResolver::BuildSchemaPath
• relates to: IMPALA-9365 Run hive-server2, YARN etc on non-HDFS test envs