Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.10.0
-
None
-
ghx-label-3
Description
Tests that were seen to fail include:
- TestClientSsl.test_tls_v12
- TestClientSsl.test_wildcard_ssl
- TestClientSsl.test_wildcard_san_ssl
Example stack trace:
self = <test_client_ssl.TestClientSsl object at 0x5bf1090>
method = <bound method TestClientSsl.test_tls_v12 of <test_client_ssl.TestClientSsl object at 0x5bf1090>>
def setup_method(self, method):
cluster_args = list()
for arg in [IMPALAD_ARGS, STATESTORED_ARGS, CATALOGD_ARGS]:
if arg in method.func_dict:
cluster_args.append("--%s=\"%s\" " % (arg, method.func_dict[arg]))
# Start a clean new cluster before each test
> self._start_impala_cluster(cluster_args)
common/custom_cluster_test_suite.py:103:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
common/custom_cluster_test_suite.py:129: in _start_impala_cluster
check_call(cmd + options, close_fds=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
popenargs = (['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_si...cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', ...],)
kwargs = {'close_fds': True}, retcode = 1
cmd = ['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_siz...a-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', ...]
def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.
The arguments are the same as for the Popen constructor. Example:
check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '['/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', '--state_store_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ', '--catalogd_args="--ssl_server_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.pem --ssl_private_key=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcard-cert.key --ssl_minimum_version=tlsv1.2 --ssl_client_ca_certificate=/data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/testutil/wildcardCA.pem --ssl_cipher_list=AES128-GCM-SHA256 " ']' returned non-zero exit status 1
Standard error output:
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000 MainThread: Debug webpage not yet available. MainThread: Waiting for num_known_live_backends=3. Current value: 0 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000 MainThread: Waiting for num_known_live_backends=3. Current value: 0 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000 MainThread: Waiting for num_known_live_backends=3. Current value: 0 MainThread: Getting num_known_live_backends from impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:25000 MainThread: Waiting for num_known_live_backends=3. Current value: 0 etc... etc...
When I look at the impalad.INFO log, this entry shows up at that timestamp:
I0818 00:43:29.000936 17712 Frontend.java:888] Compiling query: drop role grant_revoke_test_admin
I0818 00:43:29.001284 17712 Frontend.java:927] Compiled query.
I0818 00:43:29.021884 17712 impala-beeswax-server.cc:189] get_results_metadata(): query_id=114d057e65f76f3d:60e4c38f00000000
I0818 00:43:29.316861 17712 status.cc:122] Query id 114d057e65f76f3d:60e4c38f00000000 not found.
@ 0x12574e3 impala::Status::Status()
@ 0x1512d22 impala::ImpalaServer::GetExecSummary()
@ 0x158594a impala::ImpalaServer::GetExecSummary()
@ 0x1a595d5 impala::ImpalaServiceProcessor::process_GetExecSummary()
@ 0x1a56619 impala::ImpalaServiceProcessor::dispatchCall()
@ 0x11fbb96 apache::thrift::TDispatchProcessor::process()
@ 0x29076db apache::thrift::server::TThreadPoolServer::Task::run()
@ 0x28efed9 apache::thrift::concurrency::ThreadManager::Worker::run()
@ 0x13c7a6f impala::ThriftThread::RunRunnable()
@ 0x13c919b boost::_mfi::mf2<>::operator()()
@ 0x13c9031 boost::_bi::list3<>::operator()<>()
@ 0x13c8d7d boost::_bi::bind_t<>::operator()()
@ 0x13c8c90 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x13d68ba boost::function0<>::operator()()
@ 0x16a1b3d impala::Thread::SuperviseThread()
@ 0x16aa50c boost::_bi::list4<>::operator()<>()
@ 0x16aa44f boost::_bi::bind_t<>::operator()()
@ 0x16aa412 boost::detail::thread_data<>::run()
@ 0x1b8606a thread_proxy
@ 0x7f8d4de4edf3 start_thread
@ 0x7f8d4db7c1ad __clone
I0818 00:43:29.318055 17712 impala-beeswax-server.cc:231] close(): query_id=114d057e65f76f3d:60e4c38f00000000
I0818 00:43:29.318068 17712 impala-server.cc:991] UnregisterQuery(): query_id=114d057e65f76f3d:60e4c38f00000000
I0818 00:43:29.318078 17712 impala-server.cc:1077] Cancel(): query_id=114d057e65f76f3d:60e4c38f00000000
I0818 00:43:29.319111 17712 impala-server.cc:1791] Connection from client ::ffff:127.0.0.1:36947 closed, closing 1 associated session(s)
I0818 00:43:29.587154 17712 status.cc:122] Session closed
@ 0x12574e3 impala::Status::Status()
@ 0x151835f impala::ImpalaServer::CloseSessionInternal()
@ 0x151e894 impala::ImpalaServer::ConnectionEnd()
@ 0x13bb2e4 impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
@ 0x29081b3 apache::thrift::server::TThreadPoolServer::Task::run()
@ 0x28efed9 apache::thrift::concurrency::ThreadManager::Worker::run()
@ 0x13c7a6f impala::ThriftThread::RunRunnable()
@ 0x13c919b boost::_mfi::mf2<>::operator()()
@ 0x13c9031 boost::_bi::list3<>::operator()<>()
@ 0x13c8d7d boost::_bi::bind_t<>::operator()()
@ 0x13c8c90 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x13d68ba boost::function0<>::operator()()
@ 0x16a1b3d impala::Thread::SuperviseThread()
@ 0x16aa50c boost::_bi::list4<>::operator()<>()
@ 0x16aa44f boost::_bi::bind_t<>::operator()()
@ 0x16aa412 boost::detail::thread_data<>::run()
@ 0x1b8606a thread_proxy
@ 0x7f8d4de4edf3 start_thread
@ 0x7f8d4db7c1ad __clone
I0818 00:43:29.596240 17731 impala-server.cc:1391] Catalog topic entry deletion: ROLE:grant_revoke_test_admin
I0818 00:43:29.596282 17731 impala-server.cc:1391] Catalog topic entry deletion: PRIVILEGE:4.server=server1
Query (id=114d057e65f76f3d:60e4c38f00000000)
DEBUG MODE WARNING: Query profile created while running a DEBUG build of Impala. Use RELEASE builds to measure query performance.
Summary
Session ID: ab4fcedc41970d96:a677076a3a30f086
Session Type: BEESWAX
Start Time: 2017-08-18 00:43:29.000313000
End Time: 2017-08-18 00:43:29.318100000
Query Type: DDL
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.10.0-SNAPSHOT DEBUG (build 68419b2814b5633fa125c456d61cd8f3316f39e9)
User: jenkins
Connected User: jenkins
Delegated User:
Network Address: ::ffff:127.0.0.1:36947
Default Db: default
Sql Statement: drop role grant_revoke_test_admin
Coordinator: impala-boost-static-burst-slave-el7-1b82.vpc.cloudera.com:22000
Query Options (non default): MT_DOP=0
DDL Type: DROP_ROLE
Query Timeline
Query submitted: 0
Planning finished: 1000012
Request finished: 21000258
First row fetched: 22000270
Unregister query: 318003909
ImpalaServer
- CatalogOpExecTimer: 15000184
- ClientFetchWaitTimer: 297003651
- InactiveTotalTime: 0
- RowMaterializationTimer: 0
- TotalTime: 0