[CASSANDRA-17674] Test failure: org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.0.x, 4.1.x
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All

Test and Documentation Plan

See comments

• Assignee: Berenguer Blasi
• Reporter: Andres de la Peña

Authors

Berenguer Blasi

• Created: 31/May/22 17:10
• Updated: 26/Sep/23 04:55

Description

The Java upgrade dtest org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage is ~68% flaky on 4.0 and ~2% flaky on trunk, at least in CircleCI:

• 4.0: https://app.circleci.com/pipelines/github/adelapena/cassandra/1622/workflows/0086c3b1-a552-4c7a-8278-2f759cee5bdf/jobs/17288
• trunk: https://app.circleci.com/pipelines/github/adelapena/cassandra/1624/workflows/c4ce2b95-998f-459b-830e-8e3fa6637e15/jobs/17293

The error for 4.0 is:

java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:227)
	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:235)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor$ThrowingRunnable.lambda$toRunnable$0(IsolatedExecutor.java:231)
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62)
	at org.apache.cassandra.distributed.impl.Instance.lambda$shutdown$28(Instance.java:810)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$null$8(IsolatedExecutor.java:114)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Caused by: java.util.concurrent.TimeoutException: org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor@6fa17524[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 218] did not terminate on time
	at org.apache.cassandra.utils.ExecutorUtils.awaitTerminationUntil(ExecutorUtils.java:107)
	at org.apache.cassandra.utils.ExecutorUtils.awaitTermination(ExecutorUtils.java:96)
	at org.apache.cassandra.utils.ExecutorUtils.shutdownNowAndWait(ExecutorUtils.java:139)
	at org.apache.cassandra.concurrent.StageManager.shutdownAndWait(StageManager.java:128)
	at org.apache.cassandra.distributed.impl.Instance.lambda$null$26(Instance.java:803)
	at org.apache.cassandra.distributed.impl.Instance.lambda$parallelRun$32(Instance.java:968)

Whereas the error for trunk is:

org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:244)
	at org.apache.cassandra.distributed.upgrade.DropCompactStorageTest.testDropCompactStorage(DropCompactStorageTest.java:49)
	Suppressed: java.lang.NullPointerException
		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:992)
		at org.apache.cassandra.gms.Gossiper.getHostId(Gossiper.java:987)
		at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2512)
		at org.apache.cassandra.service.StorageService.onChange(StorageService.java:2086)
		at org.apache.cassandra.gms.Gossiper.markAsShutdown(Gossiper.java:479)
		at org.apache.cassandra.gms.GossipShutdownVerbHandler.doVerb(GossipShutdownVerbHandler.java:37)
		at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
		at java.lang.Thread.run(Thread.java:748)

I haven't managed to reproduce it on 4.1, although more iterations could do it.

This was discovered during CASSANDRA-17213.

Issue Links

• Discovered while testing: CASSANDRA-17213 CompactStorageUpgradeTest.compactStorageUpgradeTest fails w/OOM