[CASSANDRA-18564] Test Failure: MixedModeAvailabilityV30AllOneTest.testAvailabilityCoordinatorUpgraded

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha2, 5.1
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 5.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/c9b934df67ed4ea0779e1f07cf762b67ba6362c5

Test and Documentation Plan

PR	CI
5.0	j11
trunk	j11

• Assignee: Andres de la Peña
• Reporter: Andres de la Peña

Authors

Andres de la Peña

Reviewers

Berenguer Blasi

• Created: 02/Jun/23 16:06
• Updated: 08/Nov/23 08:25
• Resolved: 11/Sep/23 15:54

Description

The JVM upgrade dtest MixedModeAvailabilityV3XAllOneTest.testAvailabilityCoordinatorUpgraded seems to be flaky at least in trunk:

junit.framework.AssertionFailedError: Error in test '4.0.11 -> [5.0]' while upgrading to '5.0'; successful upgrades []
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:348)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailability(MixedModeAvailabilityTestBase.java:154)
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.testAvailabilityCoordinatorUpgraded(MixedModeAvailabilityTestBase.java:74)
Caused by: java.lang.AssertionError: Unexpected error while reading in case write-read consistency ALL-ONE with upgraded coordinator and 2 nodes down: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
	at org.apache.cassandra.distributed.upgrade.MixedModeAvailabilityTestBase.lambda$testAvailability$6(MixedModeAvailabilityTestBase.java:145)
	at org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.run(UpgradeTestBase.java:339)
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
	at org.apache.cassandra.service.reads.ReadCallback.awaitResults(ReadCallback.java:162)
	at org.apache.cassandra.service.reads.AbstractReadExecutor.awaitResponses(AbstractReadExecutor.java:387)
	at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2124)
	at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1995)
	at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1873)
	at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1286)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:364)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:293)
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:105)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

This has failed 143 times in 500 iterations of this CircleCI run:

• https://app.circleci.com/pipelines/github/adelapena/cassandra/2927/workflows/fcd1cd60-826b-484a-8e81-d3ba640f7de9/jobs/47659/tests

The failure has also recently appeared on Jenkins too:

• https://ci-cassandra.apache.org/job/Cassandra-trunk/1585/testReport/org.apache.cassandra.distributed.upgrade/MixedModeAvailabilityV3XAllOneTest/testAvailabilityCoordinatorUpgraded__jdk11/

Given that the failure has just appeared on Jenkins and it fails relatively easily on CircleCI, it's likely that it has been broken by a very recent change.

Issue Links

• Discovered while testing: CASSANDRA-14227 Extend maximum expiration date
• is related to: CASSANDRA-18506 Deduplicate the MixedModeAvailability upgrade jvm-dtests