Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-16961

TestKRaftUpgrade system tests fail in v3.7.1 RC1

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.8.0, 3.7.1
    • None
    • None

    Description

       

       

      ================================================================================
      SESSION REPORT (ALL TESTS)
      ducktape version: 0.11.4
      session_id:       2024-06-14--003
      run time:         86 minutes 13.705 seconds
      tests run:        24
      passed:           18
      flaky:            0
      failed:           6
      ignored:          0
      ================================================================================
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   3 minutes 44.680 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   3 minutes 42.627 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   3 minutes 28.205 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   3 minutes 42.388 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 57.679 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 57.238 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 52.545 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 56.289 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 54.953 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 59.579 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   3 minutes 21.016 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
      status:     PASS
      run time:   2 minutes 56.175 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 6.505 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 8.059 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 19.250 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 33.163 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   4 minutes 19.037 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   4 minutes 51.680 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   5 minutes 1.860 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   4 minutes 50.405 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   4 minutes 48.846 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     FAIL
      run time:   4 minutes 19.672 seconds
          TimeoutError("Kafka server didn't finish startup in 120 seconds")
      Traceback (most recent call last):
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run
          data = self.run_test()
        File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 116, in test_combined_mode_upgrade
          self.run_upgrade(from_kafka_version)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
          core_test_action(*args)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
          self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
        File "/opt/kafka-dev/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 63, in perform_version_change
          self.kafka.controller_quorum.start_node(node)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 894, in start_node
          self.wait_for_start(node, monitor, timeout_sec)
        File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 898, in wait_for_start
          monitor.wait_until("Kafka\s*Server.*started", timeout_sec=timeout_sec, backoff_sec=.25,
        File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/remoteaccount.py", line 753, in wait_until
          return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % (self.offset + 1, self.log, pattern),
        File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 58, in wait_until
          raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
      ducktape.errors.TimeoutError: Kafka server didn't finish startup in 120 seconds--------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 11.294 seconds
      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT
      status:     PASS
      run time:   3 minutes 9.933 seconds
      --------------------------------------------------------------------------------
      

      Attachments

        Issue Links

          Activity

            People

              soarez Igor Soarez
              showuon Luke Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: