Details
-
Task
-
Status: Resolved
-
Critical
-
Resolution: Won't Fix
-
3.8.0
-
None
Description
To test the new, asynchronous Kafka Consumer implementation, we migrated a slew of system tests to run both the "old" and "new" implementations. KAFKA-16272 updated the system tests in connect_distributed_test.py so it could test the new consumer with Connect. However, we are not supporting Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect system tests with the new AsyncKafkaConsumer, we get errors like the following:
test_id: kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer status: FAIL run time: 6 minutes 3.899 seconds InsufficientResourcesError('Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0') Traceback (most recent call last): File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run data = self.run_test() File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test return self.test_context.function(self.test) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 919, in test_exactly_once_source consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py", line 97, in __init__ BackgroundThreadService.__init__(self, context, num_nodes) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py", line 26, in __init__ super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 107, in __init__ self.allocate_nodes() File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 217, in allocate_nodes self.nodes = self.cluster.alloc(self.cluster_spec) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py", line 54, in alloc allocated = self.do_alloc(cluster_spec) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py", line 31, in do_alloc allocated = self._available_nodes.remove_spec(cluster_spec) File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py", line 117, in remove_spec raise InsufficientResourcesError("Not enough nodes available to allocate. " + msg) ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0
The task here is to revert the changes made in KAFKA-16272 PR 15576.
Attachments
Issue Links
- causes
-
KAFKA-16988 InsufficientResourcesError in ConnectDistributedTest system test
- Resolved
- relates to
-
KAFKA-16272 Update connect_distributed_test.py to support KIP-848’s group protocol config
- In Progress