[CASSANDRA-14922] In JVM dtests need to clean up after instance shutdown

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 4.0-alpha1, 4.0
• Component/s: Test/dtest/java
• Severity: Low
• Assignee: Joey Lynch
• Reporter: Joey Lynch

Authors

Joey Lynch

Reviewers

Alex Petrov

• Created: 04/Dec/18 18:29
• Updated: 27/Aug/20 15:14
• Resolved: 09/Jan/19 14:07

Description

Currently the unit tests are failing on circleci (example one, example two) because we use a small container (medium) for unit tests by default and the in JVM dtests are leaking a few hundred megabytes of memory per test right now. This is not a big deal because the dtest runs with the larger containers continue to function fine as well as local testing as the number of in JVM dtests is not yet high enough to cause a problem with more than 2GB of available heap. However we should fix the memory leak so that going forwards we can add more in JVM dtests without worry.

I've been working with ifesdjeen to debug, and the issue appears to be unreleased Table/Keyspace metrics (screenshot showing the leak attached). I believe that we have a few potential issues that are leading to the leaks:

1. The Instance::shutdown method is not successfully cleaning up all the metrics created by the CassandraMetricsRegistry
2. The TestCluster::close method is not waiting for all the instances to finish shutting down and cleaning up before continuing on
3. I'm not sure if this is an issue assuming we clear all metrics, but TableMetrics::release does not release all the metric references (which could leak them)

I am working on a patch which shuts down everything and assures that we do not leak memory.

Issue Links

• duplicates: CASSANDRA-14946 DistributedReadWritePathTest fails in circleci
• is related to: CASSANDRA-14974 Unit test stdout capture is malfunctioning in 4.0
• relates to: CASSANDRA-14969 Clean up ThreadLocals directly instead of via reflection