Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
3.8.0
-
None
Description
StreamsCooperativeRebalanceUpgradeTest fails on system tests when upgrading from: 2.1.1, 2.2.2 and 2.3.1.
Tests that fail:
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test Class: StreamsCooperativeRebalanceUpgradeTest Method: test_upgrade_to_cooperative_rebalance Arguments: { "upgrade_from_version": "2.1.1" }
and
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test Class: StreamsCooperativeRebalanceUpgradeTest Method: test_upgrade_to_cooperative_rebalance Arguments: { "upgrade_from_version": "2.2.2" }
and
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test Class: StreamsCooperativeRebalanceUpgradeTest Method: test_upgrade_to_cooperative_rebalance Arguments: { "upgrade_from_version": "2.3.1" }
Failure for 2.1.1 is:
TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28") Traceback (most recent call last): File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run data = self.run_test() File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test return self.test_context.function(self.test) File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance self.maybe_upgrade_rolling_bounce_and_verify(processors, File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify stdout_monitor.wait_until(verify_processing_msg, File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern), File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28
Failure for 2.2.2 is:
TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5") Traceback (most recent call last): File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run data = self.run_test() File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test return self.test_context.function(self.test) File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance self.maybe_upgrade_rolling_bounce_and_verify(processors, File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify stdout_monitor.wait_until(verify_processing_msg, File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern), File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5
Failure for 2.3.1 is:
TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21") Traceback (most recent call last): File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run data = self.run_test() File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test return self.test_context.function(self.test) File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance self.maybe_upgrade_rolling_bounce_and_verify(processors, File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify stdout_monitor.wait_until(verify_processing_msg, File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern), File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21