[KAFKA-17085] Streams Cooperative Rebalance Upgrade Test fails in System Tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.8.0
• Fix Version/s: 3.8.0
• Component/s: system tests
• Assignee: Bruno Cadonna
• Reporter: Josep Prat
• Created: 05/Jul/24 09:50
• Updated: 11/Jul/24 08:41
• Resolved: 10/Jul/24 06:49

Description

StreamsCooperativeRebalanceUpgradeTest fails on system tests when upgrading from: 2.1.1, 2.2.2 and 2.3.1.

Tests that fail:

 

Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class:  StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
  "upgrade_from_version": "2.1.1"
}
 

and

 

Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class:  StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
  "upgrade_from_version": "2.2.2"
}

and

 

 

Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class:  StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
  "upgrade_from_version": "2.3.1"
}

 

Failure for 2.1.1 is:

TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28")
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
    self.maybe_upgrade_rolling_bounce_and_verify(processors,
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
    stdout_monitor.wait_until(verify_processing_msg,
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
    return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28

Failure for 2.2.2 is:

TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5")
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
    self.maybe_upgrade_rolling_bounce_and_verify(processors,
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
    stdout_monitor.wait_until(verify_processing_msg,
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
    return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5

Failure for 2.3.1 is:

TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21")
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
    self.maybe_upgrade_rolling_bounce_and_verify(processors,
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
    stdout_monitor.wait_until(verify_processing_msg,
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
    return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21

Issue Links

• links to: GitHub Pull Request #16555
• links to: GitHub Pull Request #16561