[CASSANDRA-15170] Reduce the time needed to release in-JVM dtest cluster resources after close

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha1, 4.0
• Component/s: Test/dtest/java
• Change Category: Quality Assurance
• Complexity: Normal
• Platform: All

Source Control Link

8dcaa12baa97ce870f23ff9045f968f2fa28b2cc

Test and Documentation Plan

test-only change

• Assignee: Jon Meredith
• Reporter: Jon Meredith

Authors

Jon Meredith

Reviewers

Benedict Elliott Smith

• Created: 18/Jun/19 19:38
• Updated: 27/Aug/20 15:11
• Resolved: 20/Aug/19 13:57

Description

There are a few issues that slow the in-JVM dtests from reclaiming metaspace once the cluster is closed.

IsolatedExecutor issues the shutdown on a SingleExecutorThreadPool, sometimes this thread was still running 10s after the dtest cluster was closed.  Instead, switch to a ThreadPoolExecutor with a core pool size of 0 so that the thread executing the class loader close executes sooner.

If an OutboundTcpConnection is waiting to connect() and the endpoint is not answering, it has to wait for a timeout before it exits. Instead it should check the isShutdown flag and terminate early if shutdown has been requested.

In 3.0 and above, HintsCatalog.load uses java.nio.Files.list outside of a try-with-resources construct and leaks a file handle for the directory.  This doesn't matter for normal usage, it leaks a file handle for each dtest Instance created.

On trunk, Netty global event executor threads are still running and delay GC for the instance class loader.