[KAFKA-15427] Integration tests in TS test harness detect resource leaks

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Assignee: Christo Lolov
• Reporter: Christo Lolov
• Created: 01/Sep/23 10:13
• Updated: 03/Sep/23 08:22
• Resolved: 03/Sep/23 08:19

Description

The pull request (https://github.com/apache/kafka/pull/14116) for adding the Tiered Storage test harness uncovered resource leaks as part of the build (https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14116/28/testReport/junit/org.apache.kafka.tiered.storage.integration/OffloadAndConsumeFromLeaderTest/Build___JDK_20_and_Scala_2_13___initializationError/)

This can be reproduced locally by running the following command:

./gradlew --no-parallel --max-workers 1 -PmaxParallelForks=1 storage:test --tests org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManagerTest --tests org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest --rerun

Output:

> Task :storage:testGradle Test Run :storage:test > Gradle Test Executor 3 > RemoteLogMetadataManagerTest > testRemotePartitionDeletion(RemoteLogMetadataManager) > remoteLogMetadataManager = org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@4cc76301 PASSED

Gradle Test Run :storage:test > Gradle Test Executor 3 > RemoteLogMetadataManagerTest > testRemotePartitionDeletion(RemoteLogMetadataManager) > remoteLogMetadataManager = org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@2ca47471 PASSED

Gradle Test Run :storage:test > Gradle Test Executor 3 > RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) > remoteLogMetadataManager = org.apache.kafka.server.log.remote.storage.InmemoryRemoteLogMetadataManager@ce12fbb PASSED

Gradle Test Run :storage:test > Gradle Test Executor 3 > RemoteLogMetadataManagerTest > testFetchSegments(RemoteLogMetadataManager) > remoteLogMetadataManager = org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerWrapperWithHarness@69aabcb0 PASSED

org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError failed, log available in /Users/lolovc/Documents/kafka/storage/build/reports/testOutput/org.apache.kafka.tiered.storage.integration.OffloadAndConsumeFromLeaderTest.initializationError.test.stdoutGradle Test Run :storage:test > Gradle Test Executor 3 > OffloadAndConsumeFromLeaderTest > initializationError FAILED
    org.opentest4j.AssertionFailedError: Found 2 unexpected threads during @BeforeAll: `controller-event-thread,Test worker-EventThread` ==> expected: <true> but was: <false>

... 

The point of this Jira ticket is to find the resource leak and fix it