Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
ghx-label-2
Description
In one of the exhaustive builds authorization.test_ranger.TestRanger.test_grant_revoke_with_role failed with one of the impalads stuck during startup:
Stacktrace
common/custom_cluster_test_suite.py:181: in setup_method self._start_impala_cluster(cluster_args, **kwargs) common/custom_cluster_test_suite.py:285: in _start_impala_cluster check_call(cmd + options, close_fds=True) /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/subprocess.py:190: in check_call raise CalledProcessError(retcode, cmd) E CalledProcessError: Command '['/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/bin/start-impala-cluster.py', '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args=--server-name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger --use_customized_user_groups_mapper_for_ranger ', '--state_store_args=None ', '--catalogd_args=--server-name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger --use_customized_user_groups_mapper_for_ranger ', '--impalad_args=--default_query_options=']' returned non-zero exit status 1
Standard Error
-- 2022-07-14 01:07:04,943 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests --log_level=1 '--impalad_args=--server-name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger --use_customized_user_groups_mapper_for_ranger ' '--state_store_args=None ' '--catalogd_args=--server-name=server1 --ranger_service_type=hive --ranger_app_id=impala --authorization_provider=ranger --use_customized_user_groups_mapper_for_ranger ' --impalad_args=--default_query_options= 01:07:05 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es) 01:07:05 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/statestored.INFO 01:07:05 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/catalogd.INFO 01:07:05 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad.INFO 01:07:05 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO 01:07:05 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO 01:07:08 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:09 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:10 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:11 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:12 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:13 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:14 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:15 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:16 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:17 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es) 01:07:18 MainThread: Error starting cluster Traceback (most recent call last): File "/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/bin/start-impala-cluster.py", line 840, in <module> expected_cluster_size - expected_catalog_delays) File "/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/tests/common/impala_cluster.py", line 179, in wait_until_ready self.wait_for_num_impalads(expected_num_impalads) File "/data/jenkins/workspace/impala-cdpd-master-exhaustive-release/repos/Impala/tests/common/impala_cluster.py", line 225, in wait_for_num_impalads raise RuntimeError(msg) RuntimeError: Expected 3 impalad(s), only 2 found