[KAFKA-16052] OOM in Kafka test suite

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0
• Assignee: Divij Vaidya
• Reporter: Divij Vaidya
• Created: 27/Dec/23 12:42
• Updated: 16/Oct/24 17:25
• Resolved: 17/Jun/24 00:16

Description

Problem
Our test suite is failing with frequent OOM. Discussion in the mailing list is here: https://lists.apache.org/thread/d5js0xpsrsvhgjb10mbzo9cwsy8087x4 

Setup
To find the source of leaks, I ran the :core:test build target with a single thread (see below on how to do it) and attached a profiler to it. This Jira tracks the list of action items identified from the analysis.

How to run tests using a single thread:

diff --git a/build.gradle b/build.gradle
index f7abbf4f0b..81df03f1ee 100644
--- a/build.gradle
+++ b/build.gradle
@@ -74,9 +74,8 @@ ext {
       "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
     )-  maxTestForks = project.hasProperty('maxParallelForks') ? maxParallelForks.toInteger() : Runtime.runtime.availableProcessors()
-  maxScalacThreads = project.hasProperty('maxScalacThreads') ? maxScalacThreads.toInteger() :
-      Math.min(Runtime.runtime.availableProcessors(), 8)
+  maxTestForks = 1
+  maxScalacThreads = 1
   userIgnoreFailures = project.hasProperty('ignoreFailures') ? ignoreFailures : false   userMaxTestRetries = project.hasProperty('maxTestRetries') ? maxTestRetries.toInteger() : 0
diff --git a/gradle.properties b/gradle.properties
index 4880248cac..ee4b6e3bc1 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -30,4 +30,4 @@ scalaVersion=2.13.12
 swaggerVersion=2.2.8
 task=build
 org.gradle.jvmargs=-Xmx2g -Xss4m -XX:+UseParallelGC
-org.gradle.parallel=true
+org.gradle.parallel=false 

Result of experiment
This is how the heap memory utilized looks like, starting from tens of MB to ending with 1.5GB (with spikes of 2GB) of heap being used as the test executes. Note that the total number of threads also increases but it does not correlate with sharp increase in heap memory usage. The heap dump is available at https://www.dropbox.com/scl/fi/nwtgc6ir6830xlfy9z9cu/GradleWorkerMain_10311_27_12_2023_13_37_08.hprof.zip?rlkey=ozbdgh5vih4rcynnxbatzk7ln&dl=0 

Issue Links

• is a parent of: KAFKA-16064 Improve ControllerApisTest
• links to: GitHub Pull Request #15083
• links to: GitHub Pull Request #15094

Sub-Tasks

• Fix leaked Default DirectoryService: Resolved
• Fix leaked in ControllerApiTest: Resolved
• Fix leak of ExpirationReaper-1-AlterAcls threads in :core:test: Resolved
• Upgrade mockito to 5.8.0: Resolved
• Fix leaked ApplicationShutdownHooks in EndToEndAuthorizationTests: Resolved
• Fix leak in DelayedOperationTest: Resolved
• Fix thread leaks in ReplicaManagerTest: Resolved
• Fix leak in LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest: Resolved