[CASSANDRA-17462] Test Failure: org.apache.cassandra.distributed.test.CASTest.testSuccessfulWriteBeforeRangeMovement

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Platform: All
• Assignee: Unassigned
• Reporter: Andres de la Peña
• Created: 21/Mar/22 12:47
• Updated: 21/Mar/22 17:34
• Resolved: 21/Mar/22 17:34

Description

Intermittent failure on org.apache.cassandra.distributed.test.CASTest#testSuccessfulWriteBeforeRangeMovement for trunk:

https://ci-cassandra.apache.org/job/Cassandra-trunk/1025/testReport/org.apache.cassandra.distributed.test/CASTest/testSuccessfulWriteBeforeRangeMovement/

Failed 3 times in the last 6 runs. Flakiness: 40%, Stability: 50%
Error Message
CAS operation timed out: received 1 of 2 required responses after 0 contention retries
Stacktrace
org.apache.cassandra.exceptions.CasWriteTimeoutException: CAS operation timed out: received 1 of 2 required responses after 0 contention retries
	at org.apache.cassandra.service.paxos.Paxos$MaybeFailure.markAndThrowAsTimeoutOrFailure(Paxos.java:546)
	at org.apache.cassandra.service.paxos.Paxos.begin(Paxos.java:1048)
	at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:659)
	at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:618)
	at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:307)
	at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:500)
	at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:467)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Failures can also be hit with CircleCI test multiplexer:

https://app.circleci.com/pipelines/github/adelapena/cassandra/1391/workflows/90fb3315-394c-4c52-815f-0a35c47d2552

CircleCI results show failures with a ~30% flakiness when running the test isolated.

The same test looks ok in 4.0, as suggested by Butler and this repeated Circle run.

Issue Links

• is related to: CASSANDRA-17461 Test Failure: org.apache.cassandra.distributed.test.CASTest.testConflictingWritesWithStaleRingInformation