[KAFKA-17083] KRaft Upgrade Failures in SystemTests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.8.0
• Fix Version/s: 3.8.0
• Component/s: system tests
• Assignee: Igor Soarez
• Reporter: Josep Prat
• Created: 05/Jul/24 09:44
• Updated: 08/Jul/24 08:03
• Resolved: 05/Jul/24 21:49

Description

2 System tests for "TestKRaftUpgrade are consistently failing on 3.8 in the system tests.

Module: kafkatest.tests.core.kraft_upgrade_test
Class:  TestKRaftUpgrade
Method: test_isolated_mode_upgrade
Arguments:
{
  "from_kafka_version": "dev",
  "metadata_quorum": "ISOLATED_KRAFT"
}

 

and

 

Module: kafkatest.tests.core.kraft_upgrade_test
Class:  TestKRaftUpgrade
Method: test_combined_mode_upgrade
Arguments:
{
  "from_kafka_version": "dev",
  "metadata_quorum": "COMBINED_KRAFT"
}

 

Failure for Isolated is:

RemoteCommandError({'ssh_config': {'host': 'worker15', 'hostname': '10.140.39.207', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 'hostname': 'worker15', 'ssh_hostname': '10.140.39.207', 'user': 'ubuntu', 'externally_routable_ip': '10.140.39.207', '_logger': <Logger kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.metadata_quorum=ISOLATED_KRAFT-674 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f07e2e800a0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f07e2f3d070>, '_custom_ssh_exception_checks': None}, '/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server worker15:9092,worker16:9092,worker17:9092 upgrade --metadata 3.7', 1, b'SLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n')
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 121, in test_isolated_mode_upgrade
    self.run_upgrade(from_kafka_version)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
    self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
    core_test_action(*args)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
    self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 75, in perform_version_change
    self.kafka.upgrade_metadata_version(LATEST_STABLE_METADATA_VERSION)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py", line 920, in upgrade_metadata_version
    self.run_features_command("upgrade", new_version)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py", line 930, in run_features_command
    self.nodes[0].account.ssh(cmd)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 35, in wrapper
    return method(self, *args, **kwargs)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 293, in ssh
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker15: Command '/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server worker15:9092,worker16:9092,worker17:9092 upgrade --metadata 3.7' returned non-zero exit status 1. Remote error message: b'SLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n'

Failure for combined is:

RemoteCommandError({'ssh_config': {'host': 'worker29', 'hostname': '10.140.50.210', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 'hostname': 'worker29', 'ssh_hostname': '10.140.50.210', 'user': 'ubuntu', 'externally_routable_ip': '10.140.50.210', '_logger': <Logger kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=dev.metadata_quorum=COMBINED_KRAFT-1789 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f07e2a616a0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f07e25ac7c0>, '_custom_ssh_exception_checks': None}, '/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server worker29:9092,worker4:9092,worker1:9092 upgrade --metadata 3.7', 1, b'SLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n')
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 115, in test_combined_mode_upgrade
    self.run_upgrade(from_kafka_version)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in run_upgrade
    self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 105, in run_produce_consume_validate
    core_test_action(*args)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 105, in <lambda>
    self.run_produce_consume_validate(core_test_action=lambda: self.perform_version_change(from_kafka_version))
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py", line 75, in perform_version_change
    self.kafka.upgrade_metadata_version(LATEST_STABLE_METADATA_VERSION)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py", line 920, in upgrade_metadata_version
    self.run_features_command("upgrade", new_version)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py", line 930, in run_features_command
    self.nodes[0].account.ssh(cmd)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 35, in wrapper
    return method(self, *args, **kwargs)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 293, in ssh
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker29: Command '/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server worker29:9092,worker4:9092,worker1:9092 upgrade --metadata 3.7' returned non-zero exit status 1. Remote error message: b'SLF4J: Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in [jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: Found binding in [jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\nSLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n'

Here is an instance of this failure

Issue Links

• links to: GitHub Pull Request #16533