Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 4.0.0
-
None
-
ghx-label-1
Description
Encountered this in an S3 build.
Error Message
Failed: Timeout >120.0s
Stacktrace
custom_cluster/test_concurrent_ddls.py:191: in test_concurrent_invalidate_metadata
r1.get(timeout=60)
/usr/lib64/python2.7/multiprocessing/pool.py:548: in get
self.wait(timeout)
/usr/lib64/python2.7/multiprocessing/pool.py:543: in wait
self._cond.wait(timeout)
/usr/lib64/python2.7/threading.py:362: in wait
_sleep(delay)
E Failed: Timeout >120.0s
Standard Error
01:48:26 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
01:48:26 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO
01:48:26 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO
01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
01:48:27 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
01:48:30 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:30 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:30 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:30 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
01:48:32 MainThread: Debug webpage did not become available in expected time.
01:48:32 MainThread: Waiting for num_known_live_backends=3. Current value: None
01:48:33 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:33 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:33 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:48:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:34 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:34 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:48:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:35 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:35 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:48:36 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:36 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:36 MainThread: Waiting for num_known_live_backends=3. Current value: 0
01:48:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:37 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
01:48:37 MainThread: num_known_live_backends has reached value: 3
01:48:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:37 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25001
01:48:37 MainThread: num_known_live_backends has reached value: 3
01:48:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
01:48:38 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25002
01:48:38 MainThread: num_known_live_backends has reached value: 3
01:48:38 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
DEBUG:impala_cluster:Found 3 impalad/1 statestored/1 catalogd process(es)
INFO:impala_service:Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25010
INFO:impala_service:Metric 'statestore.live-backends' has reached desired value: 4
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25000
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25001
INFO:impala_service:num_known_live_backends has reached value: 3
DEBUG:impala_service:Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f97.vpc.cloudera.com:25002
INFO:impala_service:num_known_live_backends has reached value: 3
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~ Stack of Thread-132 (140478892721920) ~~~~~~~~~~~~~~~~~~~~~
File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self._target(*self.args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 376, in _handle_results
task = get()
File "/usr/lib64/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File "/usr/lib64/python2.7/threading.py", line 339, in wait
waiter.acquire()
~~~~~~~~~~~~~~~~~~~~ Stack of Thread-131 (140478901114624) ~~~~~~~~~~~~~~~~~~~~~
File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self._target(*self.args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 335, in _handle_tasks
for taskseq, set_length in iter(taskqueue.get, None):
File "/usr/lib64/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File "/usr/lib64/python2.7/threading.py", line 339, in wait
waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-80 (140482340452096) ~~~~~~~~~~~~~~~~~~~~~
File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self._target(*self.args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/usr/lib64/python2.7/Queue.py", line 168, in get
self.not_empty.wait()
File "/usr/lib64/python2.7/threading.py", line 339, in wait
waiter.acquire()
~~~~~~~~~~~~~~~~~~~~ Stack of Thread-129 (140478917900032) ~~~~~~~~~~~~~~~~~~~~~
File "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self._target(*self.args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 113, in worker
result = (True, func(*args, **kwds))
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/custom_cluster/test_concurrent_ddls.py", line 182, in run_invalidate_metadata
self.execute_query_expect_success(tls.client, "invalidate metadata")
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 811, in wrapper
return function(*args, **kwargs)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 819, in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_test_suite.py", line 909, in __execute_query
return impalad_client.execute(query, user=user)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/common/impala_connection.py", line 205, in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 187, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 363, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 357, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 518, in __do_rpc
return rpc()
File "/data0/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/tests/beeswax/impala_beeswax.py", line 357, in <lambda>
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/shell/gen-py/beeswaxd/BeeswaxService.py", line 143, in query
return self.recv_query()
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/shell/gen-py/beeswaxd/BeeswaxService.py", line 155, in recv_query
(fname, mtype, rseqid) = iprot.readMessageBegin()
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
sz = self.readI32()
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32
buff = self.trans.readAll(4)
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz - have)
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 159, in read
self._rbuf = StringIO(self.trans.read(max(sz, self._rbuf_size)))
File "/data/jenkins/workspace/impala-cdpd-master-core-s3/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 105, in read
buff = self.handle.recv(sz)
...