[CASSANDRA-16586] Fix flaky test testAvailabilityV30ToV4 - org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV30Test

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-rc1, 4.0
• Component/s: CI
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0-rc1

Source Control Link

https://github.com/apache/cassandra/commit/27f1bdee5ecf37eda3dde6ea61a439bdda41ea0a

Test and Documentation Plan

JVM upgrade test modified, run

• Assignee: Adam Holmberg
• Reporter: David Capwell

Authors

Adam Holmberg

Reviewers

Benjamin Lerer, Ekaterina Dimitrova

• Created: 10/Apr/21 01:10
• Updated: 25/Apr/21 11:32
• Resolved: 20/Apr/21 15:51

Description

https://app.circleci.com/pipelines/github/dcapwell/cassandra/881/workflows/8e477260-ac6a-4eab-b4be-cbc048199565/jobs/5269

testAvailabilityV30ToV4 - org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV30Test

junit.framework.AssertionFailedError: Unexpected error in case QUORUM-QUORUM with not upgraded coordinator and 1 nodes down
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase$Tester.test(MixedModeAvailabilityTestBase.java:127)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.lambda$testAvailability$2(MixedModeAvailabilityTestBase.java:79)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:186)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailability(MixedModeAvailabilityTestBase.java:81)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailability(MixedModeAvailabilityTestBase.java:53)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityV30Test.testAvailabilityV30ToV4(MixedModeAvailabilityV30Test.java:39)
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.waitOn(IsolatedExecutor.java:209)
	at org.apache.cassandra.distributed.impl.IsolatedExecutor.lambda$sync$5(IsolatedExecutor.java:109)
	at org.apache.cassandra.distributed.impl.Coordinator.executeWithResult(Coordinator.java:69)
	at org.apache.cassandra.distributed.api.ICoordinator.execute(ICoordinator.java:32)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase$Tester.lambda$test$1(MixedModeAvailabilityTestBase.java:120)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase$Tester.maybeFail(MixedModeAvailabilityTestBase.java:139)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase$Tester.test(MixedModeAvailabilityTestBase.java:119)
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 1 responses.
	at org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:136)
	at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:142)
	at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
	at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1831)
	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1780)
	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1718)
	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1627)
	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1162)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:302)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:263)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:115)
	at org.apache.cassandra.distributed.impl.Coordinator.executeInternal(Coordinator.java:107)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:69)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83)
	at java.lang.Thread.run(Thread.java:748)