[CASSANDRA-16778] Jvm-dtest race on closing and log messages

• Type: Bug
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Fix Version/s: 4.0.x, 5.x
• Component/s: Test/dtest/java
• Bug Category: Correctness
• Severity: Low
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Assignee: Unassigned
• Reporter: Berenguer Blasi
• Created: 01/Jul/21 07:16
• Updated: 07/Mar/23 10:53

Description

Failure of IPMembershipTest:

Uncaught exceptions were thrown during test

led me to investigate the logs a bit where it seems we can be in a 'closed' state but trying to perform operations such as logging still

ERROR [RequestResponseStage-8] <main> 2021-06-30 08:18:31,211 uncaught exception in thread Thread[RequestResponseStage-8,5,node3]
java.lang.IllegalStateException: Can't load ch.qos.logback.core.status.ErrorStatus. Instance class loader is already closed.
	at org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClassInternal(InstanceClassLoader.java:93)
	at org.apache.cassandra.distributed.shared.InstanceClassLoader.loadClass(InstanceClassLoader.java:87)
	at ch.qos.logback.core.spi.ContextAwareBase.addError(ContextAwareBase.java:105)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
	at ch.qos.logback.classic.Logger.log(Logger.java:765)
	at io.netty.util.internal.logging.LocationAwareSlf4JLogger.log(LocationAwareSlf4JLogger.java:50)
	at io.netty.util.internal.logging.LocationAwareSlf4JLogger.debug(LocationAwareSlf4JLogger.java:115)
	at io.netty.buffer.PoolThreadCache.free(PoolThreadCache.java:224)
	at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:516)
	at io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache.onRemoval(PooledByteBufAllocator.java:483)
	at io.netty.util.concurrent.FastThreadLocal.remove(FastThreadLocal.java:256)
	at io.netty.util.concurrent.FastThreadLocal.removeAll(FastThreadLocal.java:67)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:32)
	at java.base/java.lang.Thread.run(Thread.java:834)

Log attached

Issue Links

• relates to: CASSANDRA-17731 Clean up ScheduledExecutors, CommitLog, and MessagingService shutdown for in-JVM dtests