[IMPALA-9571] Impala fails to start up due to exception from boost::filesystem::remove_all()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0, Impala 3.4.0
• Fix Version/s: Impala 4.0.0, Impala 3.4.0
• Component/s: Backend
• Labels: broken-build flaky
• Epic Color: ghx-label-6
• Assignee: Joe McDonnell
• Reporter: Yongzhi Chen
• Created: 30/Mar/20 15:00
• Updated: 12/Nov/21 18:07
• Resolved: 04/Apr/20 00:12

Description

In impala-asf-master-exhaustive-data-cache build,
custom_cluster.test_local_catalog.TestCompactCatalogUpdates.test_restart_catalogd test failed:

Error Message
test setup failure
Stacktrace
common/custom_cluster_test_suite.py:190: in setup_method
self._start_impala_cluster(cluster_args, **kwargs)
common/custom_cluster_test_suite.py:307: in _start_impala_cluster
check_call(cmd + options, close_fds=True)
/usr/lib64/python2.7/subprocess.py:542: in check_call
raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '['/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/bin/start-impala-cluster.py', '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args=--use_local_catalog=true ', '--state_store_args=None ', '--catalogd_args=--catalog_topic_mode=minimal ', '--impalad_args=--default_query_options=']' returned non-zero exit status 1

.....

11:08:37 MainThread: Error starting cluster
Traceback (most recent call last):
File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/bin/start-impala-cluster.py", line 770, in <module>
expected_cluster_size - expected_catalog_delays)
File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/tests/common/impala_cluster.py", line 186, in wait_until_ready
early_abort_fn=check_processes_still_running)
File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/tests/common/impala_service.py", line 267, in wait_for_num_known_live_backends
early_abort_fn()
File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/tests/common/impala_cluster.py", line 178, in check_processes_still_running
assert len(self.impalads) >= expected_num_impalads
AssertionError
DEBUG:impala_cluster:Found 2 impalad/1 statestored/1 catalogd process(es)

Details:
https://master-02.jenkins.cloudera.com/job/impala-asf-master-exhaustive-data-cache/113/testReport/custom_cluster.test_local_catalog/TestCompactCatalogUpdates/test_restart_catalogd/