Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
ghx-label-1
Description
Failed test: custom_cluster.test_executor_groups.TestExecutorGroups.test_admission_control_with_multiple_coords
Error Message
AssertionError: Metric admission-controller.agg-num-running.default-pool did not reach value 1 in 30s. Dumping debug webpages in JSON format... Dumped memz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json Dumped metrics JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json Dumped queries JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json Dumped sessions JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json Dumped threadz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json Dumped rpcz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json Dumping minidumps for impalads/catalogds... Dumped minidump for Impalad PID 23807 Dumped minidump for Impalad PID 23810 Dumped minidump for Impalad PID 24897 Dumped minidump for Impalad PID 24900 Dumped minidump for Catalogd PID 23746
Stacktrace
custom_cluster/test_executor_groups.py:579: in test_admission_control_with_multiple_coords
"admission-controller.agg-num-running.default-pool", 1, timeout=30)
common/impala_service.py:143: in wait_for_metric_value
self.__metric_timeout_assert(metric_name, expected_value, timeout)
common/impala_service.py:210: in __metric_timeout_assert
assert 0, assert_string
E AssertionError: Metric admission-controller.agg-num-running.default-pool did not reach value 1 in 30s.
E Dumping debug webpages in JSON format...
E Dumped memz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/memz.json
E Dumped metrics JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/metrics.json
E Dumped queries JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/queries.json
E Dumped sessions JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/sessions.json
E Dumped threadz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/threadz.json
E Dumped rpcz JSON to $IMPALA_HOME/logs/metric_timeout_diags_20210826_05:47:13/json/rpcz.json
E Dumping minidumps for impalads/catalogds...
E Dumped minidump for Impalad PID 23807
E Dumped minidump for Impalad PID 23810
E Dumped minidump for Impalad PID 24897
E Dumped minidump for Impalad PID 24900
E Dumped minidump for Catalogd PID 23746
Standard Error
-- 2021-08-26 05:46:20,996 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=1 --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests --log_level=1 --use_exclusive_coordinators '--impalad_args= -executor_groups=coordinator ' --impalad_args=--default_query_options= 05:46:21 MainThread: Starting impala cluster without executors 05:46:21 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 05:46:21 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO 05:46:21 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 05:46:21 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO 05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 05:46:24 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 05:46:24 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:24 MainThread: 'backends' 05:46:24 MainThread: Waiting for num_known_live_backends=1. Current value: None 05:46:25 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 05:46:25 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:25 MainThread: Waiting for num_known_live_backends=1. Current value: 0 05:46:26 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 05:46:26 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:26 MainThread: Waiting for num_known_live_backends=1. Current value: 0 05:46:27 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) 05:46:27 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:27 MainThread: num_known_live_backends has reached value: 1 05:46:28 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 0 executors). -- 2021-08-26 05:46:28,337 DEBUG MainThread: Found 1 impalad/1 statestored/1 catalogd process(es) -- 2021-08-26 05:46:28,337 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010 -- 2021-08-26 05:46:28,341 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 2 -- 2021-08-26 05:46:28,341 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:28,343 INFO MainThread: num_known_live_backends has reached value: 1 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; -- connecting to: localhost:21000 -- connecting to localhost:21050 with impyla -- 2021-08-26 05:46:28,489 INFO MainThread: Closing active operation -- connecting to localhost:28000 with impyla -- 2021-08-26 05:46:28,509 INFO MainThread: Closing active operation -- 2021-08-26 05:46:28,511 INFO MainThread: Adding a coordinator -- 2021-08-26 05:46:28,511 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=2 --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests --log_level=1 --use_exclusive_coordinators --impalad_args=-executor_groups=coordinator --impalad_args=-default_pool_mem_limit=100g --impalad_args=--default_query_options= 05:46:28 MainThread: Starting impala cluster without executors 05:46:29 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 05:46:29 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/statestored.INFO 05:46:29 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 05:46:29 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad.INFO 05:46:29 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO 05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:32 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:32 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:32 MainThread: 'backends' 05:46:32 MainThread: Waiting for num_known_live_backends=2. Current value: None 05:46:33 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:33 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:33 MainThread: Waiting for num_known_live_backends=2. Current value: 0 05:46:34 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:34 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:34 MainThread: Waiting for num_known_live_backends=2. Current value: 0 05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:35 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:35 MainThread: num_known_live_backends has reached value: 2 05:46:35 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:35 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001 05:46:35 MainThread: num_known_live_backends has reached value: 2 05:46:36 MainThread: Impala Cluster Running with 2 nodes (2 coordinators, 0 executors). -- 2021-08-26 05:46:36,408 DEBUG MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) -- 2021-08-26 05:46:36,409 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010 -- 2021-08-26 05:46:36,412 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 3 -- 2021-08-26 05:46:36,412 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:36,414 INFO MainThread: num_known_live_backends has reached value: 2 -- 2021-08-26 05:46:36,414 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001 -- 2021-08-26 05:46:36,416 INFO MainThread: num_known_live_backends has reached value: 2 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; -- connecting to: localhost:21001 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; -- connecting to: localhost:21000 -- 2021-08-26 05:46:36,417 ERROR MainThread: Failed to open transport (tries_left=3) Traceback (most recent call last): File "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/hiveserver2.py", line 1099, in _execute return func(request) File "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 206, in CloseSession return self.recv_CloseSession() File "/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/impala/_thrift_gen/TCLIService/TCLIService.py", line 218, in recv_CloseSession (fname, mtype, rseqid) = iprot.readMessageBegin() File "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin sz = self.readI32() File "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32 buff = self.trans.readAll(4) File "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 60, in readAll chunk = self.read(sz - have) File "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 162, in read self.__rbuf = BufferIO(self.__trans.read(max(sz, self.__rbuf_size))) File "/data/jenkins/workspace/impala-asf-master-core-ubsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.11.0-p4/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 132, in read message='TSocket read 0 bytes') TTransportException: TSocket read 0 bytes -- connecting to localhost:21050 with impyla -- 2021-08-26 05:46:36,563 INFO MainThread: Closing active operation -- connecting to localhost:28000 with impyla -- 2021-08-26 05:46:36,585 INFO MainThread: Closing active operation -- 2021-08-26 05:46:36,586 INFO MainThread: Adding 2 executors to group default-pool-group1 with minimum size 2 -- 2021-08-26 05:46:36,586 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=2 --num_coordinators=0 --log_dir=/data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests --log_level=1 --add_executors --impalad_args=-admission_control_slots=2 --impalad_args=-executor_groups=default-pool-group1:2 --impalad_args=-mem_limit=4g --impalad_args=--default_query_options= 05:46:37 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 05:46:37 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO 05:46:37 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/impalad_node3.INFO 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) 05:46:40 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 05:46:40 MainThread: num_known_live_backends has reached value: 4 05:46:40 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) 05:46:40 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001 05:46:40 MainThread: num_known_live_backends has reached value: 4 05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) 05:46:41 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002 05:46:41 MainThread: num_known_live_backends has reached value: 4 05:46:41 MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) 05:46:41 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003 05:46:41 MainThread: num_known_live_backends has reached value: 4 05:46:41 MainThread: Impala Cluster Running with 2 nodes (0 coordinators, 2 executors). -- 2021-08-26 05:46:41,307 DEBUG MainThread: Found 4 impalad/1 statestored/1 catalogd process(es) -- 2021-08-26 05:46:41,307 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25010 -- 2021-08-26 05:46:41,310 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 5 -- 2021-08-26 05:46:41,310 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:41,312 INFO MainThread: num_known_live_backends has reached value: 4 -- 2021-08-26 05:46:41,312 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25001 -- 2021-08-26 05:46:41,314 INFO MainThread: num_known_live_backends has reached value: 4 -- 2021-08-26 05:46:41,314 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25002 -- 2021-08-26 05:46:41,316 INFO MainThread: num_known_live_backends has reached value: 4 -- 2021-08-26 05:46:41,316 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25003 -- 2021-08-26 05:46:41,318 INFO MainThread: num_known_live_backends has reached value: 4 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; SET mt_dop=2; -- executing async: localhost:21001 select * from functional_parquet.alltypes where month < 3 and id + random() < sleep(100);; -- 2021-08-26 05:46:42,208 INFO MainThread: Started query db4bd9b9ff7787ec:963015e700000000 -- 2021-08-26 05:46:42,208 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:42,213 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: None -- 2021-08-26 05:46:42,213 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:43,214 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:43,218 INFO MainThread: Metric 'admission-controller.agg-num-running.default-pool' has reached desired value: 1 -- executing async: localhost:21000 select count(*) from functional.alltypes where month + random() < 3; -- 2021-08-26 05:46:43,673 INFO MainThread: Started query de46a56beb40a1c9:6f12b84c00000000 -- 2021-08-26 05:46:43,673 INFO MainThread: Getting metric: admission-controller.local-num-queued.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:43,677 INFO MainThread: Metric 'admission-controller.local-num-queued.default-pool' has reached desired value: 1 -- getting runtime profile operation: <tests.common.impala_connection.OperationHandle object at 0x7f3651f12710> -- closing query for operation handle: <tests.common.impala_connection.OperationHandle object at 0x7f3651f12710> -- closing query for operation handle: <tests.common.impala_connection.OperationHandle object at 0x7f3651f3ae50> -- 2021-08-26 05:46:43,680 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:43,684 INFO MainThread: Metric 'admission-controller.agg-num-running.default-pool' has reached desired value: 0 SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; SET client_identifier=custom_cluster/test_executor_groups.py::TestExecutorGroups::()::test_admission_control_with_multiple_coords; SET mem_limit=4g; -- executing async: localhost:21001 select * from functional_parquet.alltypes where month < 3 and id + random() < sleep(100);; -- 2021-08-26 05:46:43,701 INFO MainThread: Started query 1841bd63d1384bff:7296726f00000000 -- 2021-08-26 05:46:43,701 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:43,705 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:43,706 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:44,707 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:44,711 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:44,711 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:45,712 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:45,717 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:45,717 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:46,717 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:46,722 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:46,722 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:47,723 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:47,727 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:47,727 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:48,728 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:48,733 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:48,733 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:49,734 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:49,739 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:49,739 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:50,740 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:50,744 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:50,744 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:51,745 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:51,749 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:51,749 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:52,750 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:52,754 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:52,754 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:53,755 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:53,760 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:53,760 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:54,760 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:54,765 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:54,765 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:55,766 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:55,770 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:55,770 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:56,771 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:56,776 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:56,776 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:57,776 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:57,780 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:57,781 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:58,781 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:58,785 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:58,785 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:46:59,786 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:46:59,790 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:46:59,790 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:00,791 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:00,796 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:00,796 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:01,797 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:01,801 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:01,801 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:02,802 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:02,807 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:02,807 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:03,808 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:03,813 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:03,813 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:04,814 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:04,818 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:04,818 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:05,819 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:05,823 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:05,823 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:06,824 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:06,828 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:06,828 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:07,829 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:07,834 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:07,834 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:08,835 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:08,839 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:08,839 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:09,840 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:09,845 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:09,845 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:10,845 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:10,850 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:10,850 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:11,851 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:11,855 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:11,855 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:12,857 INFO MainThread: Getting metric: admission-controller.agg-num-running.default-pool from impala-ec2-centos74-m5-4xlarge-ondemand-1f45.vpc.cloudera.com:25000 -- 2021-08-26 05:47:12,861 INFO MainThread: Waiting for metric value 'admission-controller.agg-num-running.default-pool'=1. Current value: 0 -- 2021-08-26 05:47:12,861 INFO MainThread: Sleeping 1s before next retry. -- 2021-08-26 05:47:13,862 INFO MainThread: Metric admission-controller.agg-num-running.default-pool did not reach value 1 in 30s. Failing...