Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 4.4.0
-
None
-
ghx-label-4
Description
We have plenty of custom_cluster tests that assert against content of Impala daemon log files while the process is still running using assert_log_contains() and it's wrappers. The method specifically mention about disabling glog buffering ('-logbuflevel=-1'), but not all custom_cluster tests do that.
def assert_log_contains(self, daemon, level, line_regex, expected_count=1, timeout_s=6, dry_run=False): """ Assert that the daemon log with specified level (e.g. ERROR, WARNING, INFO) contains expected_count lines with a substring matching the regex. When expected_count is -1, at least one match is expected. Retries until 'timeout_s' has expired. The default timeout is the default minicluster log buffering time (5 seconds) with a one second buffer. When using this method to check log files of running processes, the caller should make sure that log buffering has been disabled, for example by adding '-logbuflevel=-1' to the daemon startup options or set timeout_s to a value higher than the log flush interval. Returns the result of the very last call to line_regex.search or None if expected_count is 0 or the line_regex did not match any lines. """
This often result in flaky test that hard to triage and often neglected if it does not frequently run in core exploration.
We can improve this by adding boolean param into CustomClusterTestSuite.with_args, say 'disable_log_buffering', for test to declare intention to inspect log files in live minicluster. If it is True, start minicluster with '-logbuflevel=-1' for all daemons. If it is False, log WARNING on any calls to assert_log_contains().