Regression custom_cluster.test_admission_controller.TestAdmissionController.test_queue_reasons_slots (from pytest) Failing for the past 1 build (Since Failed#673 ) Took 1 min 58 sec. add description Error Message Timeout: query 504b3a2511f3cd0e:e27bec6b00000000 did not reach one of the expected states [4], last known state 3 Stacktrace custom_cluster/test_admission_controller.py:967: in test_queue_reasons_slots TIMEOUT_S, config_options={"mt_dop": 4}) custom_cluster/test_admission_controller.py:277: in _execute_and_collect_profiles state = self.wait_for_any_state(handle, expected_states, timeout_s) common/impala_test_suite.py:1081: in wait_for_any_state actual_state)) E Timeout: query 504b3a2511f3cd0e:e27bec6b00000000 did not reach one of the expected states [4], last known state 3 Standard Error -- 2020-11-16 13:24:50,354 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests --log_level=1 '--impalad_args=-vmodule admission-controller=3 -default_pool_max_requests 100 -default_pool_max_queued 10 -default_pool_mem_limit -1 -admission_control_slots=4 -executor_groups=default-pool-group1 ' '--state_store_args=-statestore_heartbeat_frequency_ms=100 -statestore_priority_update_frequency_ms=100 ' --impalad_args=--default_query_options= 13:24:50 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 13:24:50 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/statestored.INFO 13:24:50 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 13:24:50 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad.INFO 13:24:50 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO 13:24:50 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO 13:24:53 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:24:53 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:24:53 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 13:24:54 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused')) 13:24:56 MainThread: Debug webpage did not become available in expected time. 13:24:56 MainThread: Waiting for num_known_live_backends=3. Current value: None 13:24:57 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:24:57 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 13:24:57 MainThread: Waiting for num_known_live_backends=3. Current value: 0 13:24:58 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:24:58 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 13:24:58 MainThread: Waiting for num_known_live_backends=3. Current value: 0 13:24:59 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:24:59 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 13:24:59 MainThread: Waiting for num_known_live_backends=3. Current value: 0 13:25:00 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:25:00 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 13:25:00 MainThread: num_known_live_backends has reached value: 3 13:25:00 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:25:00 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25001 13:25:00 MainThread: num_known_live_backends has reached value: 3 13:25:01 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 13:25:01 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25002 13:25:01 MainThread: num_known_live_backends has reached value: 3 13:25:01 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors). -- 2020-11-16 13:25:01,769 DEBUG MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) -- 2020-11-16 13:25:01,770 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25010 -- 2020-11-16 13:25:01,771 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com -- 2020-11-16 13:25:01,773 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4 -- 2020-11-16 13:25:01,773 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25000 -- 2020-11-16 13:25:01,774 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com -- 2020-11-16 13:25:01,775 INFO MainThread: num_known_live_backends has reached value: 3 -- 2020-11-16 13:25:01,775 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25001 -- 2020-11-16 13:25:01,775 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com -- 2020-11-16 13:25:01,776 INFO MainThread: num_known_live_backends has reached value: 3 -- 2020-11-16 13:25:01,776 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:25002 -- 2020-11-16 13:25:01,777 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com -- 2020-11-16 13:25:01,778 INFO MainThread: num_known_live_backends has reached value: 3 SET client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_queue_reasons_slots; -- connecting to: localhost:21000 -- connecting to localhost:21050 with impyla -- 2020-11-16 13:25:01,919 INFO MainThread: Closing active operation -- connecting to localhost:28000 with impyla -- 2020-11-16 13:25:01,940 INFO MainThread: Closing active operation -- 2020-11-16 13:25:01,942 INFO MainThread: Starting new HTTP connection (1): localhost SET client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_queue_reasons_slots; -- connecting to: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 SET client_identifier=custom_cluster/test_admission_controller.py::TestAdmissionController::()::test_queue_reasons_slots; SET mt_dop=4; -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 select min(ss_wholesale_cost) from tpcds_parquet.store_sales; -- 2020-11-16 13:25:07,945 INFO MainThread: Started query aa4a301189a4d144:fdc17ff700000000 -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 select min(ss_wholesale_cost) from tpcds_parquet.store_sales; -- 2020-11-16 13:25:08,089 INFO MainThread: Started query 554d88d8f812e22d:efbaf75200000000 -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 select min(ss_wholesale_cost) from tpcds_parquet.store_sales; -- 2020-11-16 13:25:08,246 INFO MainThread: Started query 5b470ebea7782154:ea22bdb000000000 -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 select min(ss_wholesale_cost) from tpcds_parquet.store_sales; -- 2020-11-16 13:25:08,445 INFO MainThread: Started query 0a4232658ea48bc5:c015646900000000 -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000 select min(ss_wholesale_cost) from tpcds_parquet.store_sales; -- 2020-11-16 13:25:08,568 INFO MainThread: Started query 504b3a2511f3cd0e:e27bec6b00000000 -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- fetching results from: -- getting runtime profile operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- fetching results from: -- getting runtime profile operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- fetching results from: -- getting runtime profile operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- fetching results from: -- getting runtime profile operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- getting state for operation: -- closing connection to: impala-ec2-centos74-m5-4xlarge-ondemand-018e.vpc.cloudera.com:21000