[KAFKA-16988] InsufficientResourcesError in ConnectDistributedTest system test

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.8.0, 3.7.1
• Assignee: Luke Chen
• Reporter: Luke Chen
• Created: 18/Jun/24 02:50
• Updated: 18/Jun/24 17:55
• Resolved: 18/Jun/24 12:14

Description

Saw InsufficientResourcesError when running `ConnectDistributedTest#test_exactly_once_source` system test.

 

<testcase name="test_exactly_once_source_clean=False_connect_protocol=compatible_metadata_quorum=ZK_use_new_coordinator=False" classname="kafkatest.tests.connect.connect_distributed_test" time="403.812"><error message="InsufficientResourcesError('linux nodes requested: 1. linux nodes available: 0')" type="exception">InsufficientResourcesError('linux nodes requested: 1. linux nodes available: 0') Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run data = self.run_test() File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test return self.test_context.function(self.test) File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/opt/kafka-dev/tests/kafkatest/tests/connect/connect_distributed_test.py", line 928, in test_exactly_once_source consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True) File "/opt/kafka-dev/tests/kafkatest/services/console_consumer.py", line 97, in __init__ BackgroundThreadService.__init__(self, context, num_nodes) File "/usr/local/lib/python3.9/dist-packages/ducktape/services/background_thread.py", line 26, in __init__ super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line 107, in __init__ self.allocate_nodes() File "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line 217, in allocate_nodes self.nodes = self.cluster.alloc(self.cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/cluster.py", line 54, in alloc allocated = self.do_alloc(cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/finite_subcluster.py", line 37, in do_alloc good_nodes, bad_nodes = self._available_nodes.remove_spec(cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/node_container.py", line 131, in remove_spec raise InsufficientResourcesError(err) ducktape.cluster.node_container.InsufficientResourcesError: linux nodes requested: 1. linux nodes available: 0 </error>

<testcase name="test_exactly_once_source_clean=False_connect_protocol=sessioned_metadata_quorum=ZK_use_new_coordinator=False" classname="kafkatest.tests.connect.connect_distributed_test" time="376.160"><error message="InsufficientResourcesError('linux nodes requested: 1. linux nodes available: 0')" type="exception">InsufficientResourcesError('linux nodes requested: 1. linux nodes available: 0') Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 186, in _do_run data = self.run_test() File "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 246, in run_test return self.test_context.function(self.test) File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/opt/kafka-dev/tests/kafkatest/tests/connect/connect_distributed_test.py", line 928, in test_exactly_once_source consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True) File "/opt/kafka-dev/tests/kafkatest/services/console_consumer.py", line 97, in __init__ BackgroundThreadService.__init__(self, context, num_nodes) File "/usr/local/lib/python3.9/dist-packages/ducktape/services/background_thread.py", line 26, in __init__ super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line 107, in __init__ self.allocate_nodes() File "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line 217, in allocate_nodes self.nodes = self.cluster.alloc(self.cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/cluster.py", line 54, in alloc allocated = self.do_alloc(cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/finite_subcluster.py", line 37, in do_alloc good_nodes, bad_nodes = self._available_nodes.remove_spec(cluster_spec) File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/node_container.py", line 131, in remove_spec raise InsufficientResourcesError(err) ducktape.cluster.node_container.InsufficientResourcesError: linux nodes requested: 1. linux nodes available: 0 </error>
...

 

Issue Links

• is caused by: KAFKA-16272 Update connect_distributed_test.py to support KIP-848’s group protocol config
• is caused by: KAFKA-16578 Revert changes to connect_distributed_test.py for the new async Consumer
• links to: GitHub Pull Request #16379