[CASSANDRA-18944] Test failure: org.apache.cassandra.simulator.test.ShortPaxosSimulationTest.simulationTest

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 5.0-rc1, 5.0, 5.1
• Component/s: Test/unit
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Assignee: Unassigned
• Reporter: Andres de la Peña
• Created: 19/Oct/23 12:08
• Updated: 04/Jan/24 18:27
• Resolved: 30/Nov/23 15:05

Description

The simulator test org.apache.cassandra.simulator.test.ShortPaxosSimulationTest#simulationTest is flaky on 5.0 and trunk:

• https://app.circleci.com/pipelines/github/mike-tr-adamson/cassandra/332/workflows/946e28f4-2dec-4384-ac38-de011093f6c6/jobs/25735/tests
• https://app.circleci.com/pipelines/github/adelapena/cassandra/3253/workflows/e48b49e9-cf36-412a-a811-d813031e6f01/jobs/83735/tests
• https://app.circleci.com/pipelines/github/adelapena/cassandra/3254/workflows/69f451ef-fb39-48e4-b1d1-40ee4141b0c1/jobs/83739/tests

org.apache.cassandra.simulator.SimulationException: Failed on seed 0xb01206bb3b021127
Caused by: java.lang.ClassCastException: class org.apache.cassandra.net.NoPayload cannot be cast to class org.apache.cassandra.gms.GossipShutdown (org.apache.cassandra.net.NoPayload and org.apache.cassandra.gms.GossipShutdown are in unnamed module of loader org.apache.cassandra.distributed.shared.InstanceClassLoader @68801feb)
	at org.apache.cassandra.gms.GossipShutdown$Serializer.serialize(GossipShutdown.java:41)
	at org.apache.cassandra.net.Message$Serializer.serialize(Message.java:722)
	at org.apache.cassandra.distributed.impl.Instance.serializeMessage(Instance.java:427)
	at org.apache.cassandra.distributed.impl.Instance.lambda$registerOutboundFilter$5(Instance.java:370)
	at org.apache.cassandra.net.OutboundSink$Filtered.accept(OutboundSink.java:54)
	at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:449)
	at org.apache.cassandra.net.MessagingService.send(MessagingService.java:419)
	at org.apache.cassandra.gms.Gossiper.unsafeSendShutdown(Gossiper.java:2634)
	at org.apache.cassandra.simulator.cluster.OnInstanceSendShutdown.lambda$invokableSendShutdown$1(OnInstanceSendShutdown.java:48)
	at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
	at org.apache.cassandra.concurrent.SyncFutureTask$1.call(SyncFutureTask.java:46)
	at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68)
	at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:584)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

The test failure can easily be reproduced on CircleCI with:

.circleci/generate.sh -sp \
  -e REPEATED_SIMULATOR_DTESTS=org.apache.cassandra.simulator.test.ShortPaxosSimulationTest \
  -e REPEATED_SIMULATOR_DTESTS_COUNT=100

Flakiness seems ~18%.

The test failure is not reported on Butler because simulator tests are not run by Jenkins.

Issue Links

• Discovered while testing: CASSANDRA-18905 Index.Group is incorrectly unregistered from the SecondaryIndexManager
• is fixed by: CASSANDRA-18952 Add metrics and logging to repair retries