-- 2024-01-08 17:27:31,199 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests --log_level=1 '--state_store_args=None ' '--catalogd_args=--hms_event_polling_interval_s=1 ' --impalad_args=--default_query_options= 17:27:31 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 17:27:31 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests/statestored.INFO 17:27:31 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 17:27:31 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests/impalad.INFO 17:27:31 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO 17:27:31 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-release-arm/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO 17:27:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:34 MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 17:27:34 MainThread: Waiting for num_known_live_backends=3. Current value: 0 17:27:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:35 MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 17:27:35 MainThread: Waiting for num_known_live_backends=3. Current value: 0 17:27:36 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:36 MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 17:27:36 MainThread: num_known_live_backends has reached value: 3 17:27:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:37 MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25001 17:27:37 MainThread: num_known_live_backends has reached value: 3 17:27:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) 17:27:38 MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25002 17:27:38 MainThread: num_known_live_backends has reached value: 3 17:27:38 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors). -- 2024-01-08 17:27:38,642 DEBUG MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) -- 2024-01-08 17:27:38,643 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25010 -- 2024-01-08 17:27:38,645 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4 -- 2024-01-08 17:27:38,645 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 -- 2024-01-08 17:27:38,647 INFO MainThread: num_known_live_backends has reached value: 3 -- 2024-01-08 17:27:38,647 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25001 -- 2024-01-08 17:27:38,649 INFO MainThread: num_known_live_backends has reached value: 3 -- 2024-01-08 17:27:38,649 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25002 -- 2024-01-08 17:27:38,650 INFO MainThread: num_known_live_backends has reached value: 3 SET client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_iceberg_self_events; -- connecting to: localhost:21000 -- 2024-01-08 17:27:38,651 INFO MainThread: Could not connect to ('::1', 21000, 0, 0) Traceback (most recent call last): File "/data/jenkins/workspace/impala-asf-master-core-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 137, in open handle.connect(sockaddr) File "/data/jenkins/workspace/impala-asf-master-core-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 111] Connection refused -- connecting to localhost:21050 with impyla -- 2024-01-08 17:27:38,651 INFO MainThread: Could not connect to ('::1', 21050, 0, 0) Traceback (most recent call last): File "/data/jenkins/workspace/impala-asf-master-core-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/thrift-0.16.0-p6/python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 137, in open handle.connect(sockaddr) File "/data/jenkins/workspace/impala-asf-master-core-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) error: [Errno 111] Connection refused -- 2024-01-08 17:27:38,822 INFO MainThread: Closing active operation -- connecting to localhost:28000 with impyla -- 2024-01-08 17:27:38,840 INFO MainThread: Closing active operation SET client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_iceberg_self_events; SET sync_ddl=False; -- executing against localhost:21000 DROP DATABASE IF EXISTS `test_iceberg_self_events_aa5b394f` CASCADE; -- 2024-01-08 17:27:38,896 INFO MainThread: Started query 094aaadd2ecd116b:baab225f00000000 SET client_identifier=custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_iceberg_self_events; SET sync_ddl=False; -- executing against localhost:21000 CREATE DATABASE `test_iceberg_self_events_aa5b394f`; -- 2024-01-08 17:27:39,017 INFO MainThread: Started query e6445cde9f055278:fc9f34bc00000000 -- 2024-01-08 17:27:39,169 INFO MainThread: Created database "test_iceberg_self_events_aa5b394f" for test ID "custom_cluster/test_events_custom_configs.py::TestEventProcessingCustomConfigs::()::test_iceberg_self_events" -- executing against localhost:21000 CREATE TABLE test_iceberg_self_events_aa5b394f.test_iceberg_events (i int) STORED AS ICEBERG TBLPROPERTIES ('iceberg.catalog'='hadoop.tables'); -- 2024-01-08 17:27:39,235 INFO MainThread: Started query 00473600615b655b:2b3c803900000000 -- executing against localhost:21000 INSERT OVERWRITE test_iceberg_self_events_aa5b394f.test_iceberg_events VALUES (1); -- 2024-01-08 17:27:41,167 INFO MainThread: Started query 574d87ea986c59ce:9bdfcb6e00000000 -- 2024-01-08 17:27:41,926 INFO MainThread: Waiting until events processor syncs to event id:60357 -- 2024-01-08 17:27:41,932 DEBUG MainThread: Metric last-synced-event-id has reached the desired value: 60357 -- 2024-01-08 17:27:41,969 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 -- 2024-01-08 17:27:41,972 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1851 -- 2024-01-08 17:27:41,972 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25001 -- 2024-01-08 17:27:41,985 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1851 -- 2024-01-08 17:27:41,985 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25002 -- 2024-01-08 17:27:41,998 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1851 -- executing against localhost:21000 ALTER TABLE test_iceberg_self_events_aa5b394f.test_iceberg_events ADD COLUMN j INT; -- 2024-01-08 17:27:42,035 INFO MainThread: Started query ad431ceb72233c4a:a4b03a8400000000 -- 2024-01-08 17:27:42,190 INFO MainThread: Waiting until events processor syncs to event id:60358 -- 2024-01-08 17:27:42,299 DEBUG MainThread: Metric last-synced-event-id has reached the desired value: 60358 -- 2024-01-08 17:27:42,314 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25000 -- 2024-01-08 17:27:42,326 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1853 -- 2024-01-08 17:27:42,327 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25001 -- 2024-01-08 17:27:42,329 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1853 -- 2024-01-08 17:27:42,329 INFO MainThread: Getting metric: catalog.curr-version from impala-ec2-rhel88-m7g-4xlarge-ondemand-0e7f.vpc.cloudera.com:25002 -- 2024-01-08 17:27:42,332 INFO MainThread: Metric 'catalog.curr-version' has reached desired value: 1853