Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-17085

Streams Cooperative Rebalance Upgrade Test fails in System Tests

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.8.0
    • 3.8.0
    • system tests
    • None

    Description

      StreamsCooperativeRebalanceUpgradeTest fails on system tests when upgrading from: 2.1.1, 2.2.2 and 2.3.1.

      Tests that fail:

       

      Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
      Class:  StreamsCooperativeRebalanceUpgradeTest
      Method: test_upgrade_to_cooperative_rebalance
      Arguments:
      {
        "upgrade_from_version": "2.1.1"
      }
       
      

      and

       

      Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
      Class:  StreamsCooperativeRebalanceUpgradeTest
      Method: test_upgrade_to_cooperative_rebalance
      Arguments:
      {
        "upgrade_from_version": "2.2.2"
      }
      

      and

       

       

      Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
      Class:  StreamsCooperativeRebalanceUpgradeTest
      Method: test_upgrade_to_cooperative_rebalance
      Arguments:
      {
        "upgrade_from_version": "2.3.1"
      }
      

       

      Failure for 2.1.1 is:

      TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28")
      Traceback (most recent call last):
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
          data = self.run_test()
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
          return self.test_context.function(self.test)
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
          self.maybe_upgrade_rolling_bounce_and_verify(processors,
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
          stdout_monitor.wait_until(verify_processing_msg,
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker28

      Failure for 2.2.2 is:

      TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5")
      Traceback (most recent call last):
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
          data = self.run_test()
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
          return self.test_context.function(self.test)
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
          self.maybe_upgrade_rolling_bounce_and_verify(processors,
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
          stdout_monitor.wait_until(verify_processing_msg,
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker5

      Failure for 2.3.1 is:

      TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21")
      Traceback (most recent call last):
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
          data = self.run_test()
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
          return self.test_context.function(self.test)
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 101, in test_upgrade_to_cooperative_rebalance
          self.maybe_upgrade_rolling_bounce_and_verify(processors,
        File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py", line 182, in maybe_upgrade_rolling_bounce_and_verify
          stdout_monitor.wait_until(verify_processing_msg,
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 736, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]* records so far' message ubuntu@worker21

      Attachments

        Activity

          People

            cadonna Bruno Cadonna
            jlprat Josep Prat
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: