[CASSANDRA-17636] Test Failure: org.apache.cassandra.distributed.upgrade.MixedModeFrom3ReplicationTest.testSimpleStrategy

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-beta1, 4.1.x, 5.0-alpha1, 5.0
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Critical
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/0119beee7afb18ae1201da2d5bbbae80ce0dc545

Test and Documentation Plan

4.0 PR
4.1 PR
trunk PR

• Assignee: Jon Meredith
• Reporter: Andres de la Peña

Authors

Jon Meredith

Reviewers

David Capwell

• Created: 18/May/22 17:02
• Updated: 12/Sep/23 13:03
• Resolved: 30/Jun/22 23:23

Description

The upgrade in-JVM dtest org.apache.cassandra.distributed.upgrade.MixedModeFrom3ReplicationTest.testSimpleStrategy is flaky on 4.1 and trunk.

Jenkins failure:
https://ci-cassandra.apache.org/job/Cassandra-4.1/21/testReport/org.apache.cassandra.distributed.upgrade/MixedModeFrom3UnloggedBatchTest/testSimpleStrategy/
https://ci-cassandra.apache.org/job/Cassandra-trunk/1149/testReport/org.apache.cassandra.distributed.upgrade/MixedModeFrom3ReplicationTest/testSimpleStrategy_2/

Circle failure:
https://app.circleci.com/pipelines/github/adelapena/cassandra/1564/workflows/e8dbd50a-d574-4ef8-a741-795a93cd3e85/jobs/16725

Error Message:

Uncaught exceptions were thrown during test

Stacktrace:

org.apache.cassandra.distributed.shared.ShutdownException: Uncaught exceptions were thrown during test
	at org.apache.cassandra.distributed.impl.AbstractCluster.checkAndResetUncaughtExceptions(AbstractCluster.java:1056)
	at org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1042)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:239)
	at org.apache.cassandra.distributed.upgrade.MixedModeBatchTestBase.testSimpleStrategy(MixedModeBatchTestBase.java:78)
	at org.apache.cassandra.distributed.upgrade.MixedModeBatchTestBase.testSimpleStrategy(MixedModeBatchTestBase.java:48)
	at org.apache.cassandra.distributed.upgrade.MixedModeFrom3UnloggedBatchTest.testSimpleStrategy(MixedModeFrom3UnloggedBatchTest.java:36)
	Suppressed: java.lang.RuntimeException: java.lang.AssertionError
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
		at org.apache.cassandra.distributed.impl.Instance.lambda$null$6(Instance.java:495)
		at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
		at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:120)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)
	Caused by: java.lang.AssertionError
		at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:1171)
		at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:1185)
		at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicas(AbstractReplicationStrategy.java:95)
		at org.apache.cassandra.locator.AbstractReplicationStrategy.getLocalReplicaFor(AbstractReplicationStrategy.java:111)
		at org.apache.cassandra.db.ReadCommandVerbHandler.validateTransientStatus(ReadCommandVerbHandler.java:108)
		at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
		at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
	Suppressed: java.lang.RuntimeException: java.lang.AssertionError
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
		at org.apache.cassandra.distributed.impl.Instance.lambda$null$6(Instance.java:495)
		at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:124)
		at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:120)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)
	Caused by: java.lang.AssertionError
		at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:1171)
		at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:1185)
		at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicas(AbstractReplicationStrategy.java:95)
		at org.apache.cassandra.locator.AbstractReplicationStrategy.getLocalReplicaFor(AbstractReplicationStrategy.java:111)
		at org.apache.cassandra.db.ReadCommandVerbHandler.validateTransientStatus(ReadCommandVerbHandler.java:108)
		at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50)
		at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
		at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)

Issue Links

• causes: CASSANDRA-17641 Fix flaky dtest - org.apache.cassandra.distributed.test.SchemaTest.readRepair
• causes: CASSANDRA-17642 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.CompactStorageUpgradeTest.compactStorageImplicitNullInClusteringTest
• causes: CASSANDRA-17651 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.BatchUpgradeTest.batchTest
• causes: CASSANDRA-17652 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.MixedModeConsistencyV3XTest.testConsistency
• is related to: CASSANDRA-17307 Test Failure: org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV30Test.testAvailability
• is related to: CASSANDRA-17310 Test Failure: org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV3XTest.testAvailability
• is related to: CASSANDRA-17641 Fix flaky dtest - org.apache.cassandra.distributed.test.SchemaTest.readRepair
• is related to: CASSANDRA-17642 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.CompactStorageUpgradeTest.compactStorageImplicitNullInClusteringTest
• is related to: CASSANDRA-17651 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.BatchUpgradeTest.batchTest
• is related to: CASSANDRA-17652 Fix flaky dtest - org.apache.cassandra.distributed.upgrade.MixedModeConsistencyV3XTest.testConsistency
• relates to: CASSANDRA-17656 jvm-dtest upgrade tests run all supported pairs of upgrades between from/to but does not actually test all patches from/to