[KAFKA-17207] ConnectWorkerIntegrationTest.testRequestTimeouts times out in stop(), leaking clients

• Type: Test
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.9.0
• Component/s: connect
• Labels: newbie
• Assignee: Xuze Yang
• Reporter: Greg Harris
• Created: 26/Jul/24 19:00
• Updated: 06/Nov/24 05:06

Description

The testRequestTimeouts deletes the internal config topic, putting the connect workers into a bad state. When the test goes to clean up, it calls DistributedHerder#stop, which waits for the herder executor to stop. This times out, because the herder executor is blocked closing the KafkaConfigBackingStore's producer. This log message gets printed:

[2024-07-26 11:52:50,817] ERROR Executor java.util.concurrent.ThreadPoolExecutor@7ae97a58[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] did not terminate in time (org.apache.kafka.common.utils.ThreadUtils:83)

This effectively leaks the Kafka clients for the workers' internal topics, and the herder executor thread. Instead, either the producer should not block indefinitely on a missing topic, or the cluster state should be healed enough for the producer to shutdown cleanly.

Issue Links

• Discovered while testing: KAFKA-15845 Add Junit5 test extension which detects leaked Kafka clients and servers