[KAFKA-17084] Network Degrade Test fails in System Tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 3.8.0
• Component/s: system tests
• Assignee: Unassigned
• Reporter: Josep Prat
• Created: 05/Jul/24 09:46
• Updated: 09/Jul/24 12:11
• Resolved: 08/Jul/24 07:42

Description

Tests for NetworkDegradeTest fail consistently on the 3.8 branch.

 

Tests failing are:

 

Module: kafkatest.tests.core.network_degrade_test
Class:  NetworkDegradeTest
Method: test_latency
Arguments:
{
  "device_name": "eth0",
  "latency_ms": 50,
  "rate_limit_kbit": 1000,
  "task_name": "latency-100-rate-1000"
}

 

and

 

Module: kafkatest.tests.core.network_degrade_test
Class:  NetworkDegradeTest
Method: test_latency
Arguments:
{
  "device_name": "eth0",
  "latency_ms": 50,
  "rate_limit_kbit": 0,
  "task_name": "latency-100"
}

 

Failure for the first one is:

RemoteCommandError({'ssh_config': {'host': 'worker30', 'hostname': '10.140.34.105', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 'hostname': 'worker30', 'ssh_hostname': '10.140.34.105', 'user': 'ubuntu', 'externally_routable_ip': '10.140.34.105', '_logger': <Logger kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100-rate-1000.device_name=eth0.latency_ms=50.rate_limit_kbit=1000-1790 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f17a237dc10>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f17a2393910>, '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20 worker21', 1, b'')
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/network_degrade_test.py", line 66, in test_latency
    for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" % zk1.account.hostname):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 680, in next
    return next(self.iter_obj)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 347, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker30: Command 'ping -i 1 -c 20 worker21' returned non-zero exit status 1.

And for the second one is:

RemoteCommandError({'ssh_config': {'host': 'worker28', 'hostname': '10.140.41.79', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 'hostname': 'worker28', 'ssh_hostname': '10.140.41.79', 'user': 'ubuntu', 'externally_routable_ip': '10.140.41.79', '_logger': <Logger kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100.device_name=eth0.latency_ms=50.rate_limit_kbit=0-1791 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f17a1c7b7c0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f17a1c7b2b0>, '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20 worker27', 1, b'')
Traceback (most recent call last):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/network_degrade_test.py", line 66, in test_latency
    for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" % zk1.account.hostname):
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 680, in next
    return next(self.iter_obj)
  File "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py", line 347, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker28: Command 'ping -i 1 -c 20 worker27' returned non-zero exit status 1.