[CASSANDRA-17461] Test Failure: org.apache.cassandra.distributed.test.CASTest.testConflictingWritesWithStaleRingInformation

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-beta1, 4.1
• Component/s: Test/dtest/java
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.1-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/286d2ee053ac2e6fe50749fa2833533254697662

Test and Documentation Plan

circle

• Assignee: Blake Eggleston
• Reporter: Andres de la Peña

Authors

Blake Eggleston

Reviewers

Benedict Elliott Smith, Blake Eggleston

• Created: 21/Mar/22 12:27
• Updated: 05/Oct/22 22:22
• Resolved: 26/Sep/22 15:28

Description

Intermittent failures on org.apache.cassandra.distributed.test.CASTest for trunk:

* testConflictingWritesWithStaleRingInformation

• testSuccessfulWriteBeforeRangeMovement
• testSuccessfulWriteDuringRangeMovementFollowedByConflicting
• testSucccessfulWriteDuringRangeMovementFollowedByRead

All four seem to have the same aspect:

Failed 2 times in the last 5 runs. Flakiness: 50%, Stability: 60%
Error Message
CAS operation timed out: received 1 of 2 required responses after 0 contention retries
Stacktrace
org.apache.cassandra.exceptions.CasWriteTimeoutException: CAS operation timed out: received 1 of 2 required responses after 0 contention retries
	at org.apache.cassandra.service.paxos.Paxos$MaybeFailure.markAndThrowAsTimeoutOrFailure(Paxos.java:547)
	at org.apache.cassandra.service.paxos.Paxos.begin(Paxos.java:1048)
	at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:659)
	at org.apache.cassandra.service.paxos.Paxos.cas(Paxos.java:618)
	at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:307)
	at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:500)
	at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:467)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
	at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
	at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
	at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
	at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Standard Output
DEBUG [main] 2022-03-19 16:20:42,868 Reflections.java:198 - going to scan these urls: [jar:file:/home/cassandra/cassandra/build/apache-cassandra-4.1-SNAPSHOT.jar!/, jar:file:/home/cassandra/cassandra/build/test/lib/jars/simulator-bootstrap.jar!/, jar:file:/home/cassandra/cassandra/build/test/lib/jars/dtest-api-0.0.12.jar!/, file:/home/cassandra/cassandra/build/classes/fqltool/, file:/home/cassandra/cassandra/build/test/classes/, file:/home/cassandra/cassandra/build/classes/main/, file:/home/cass
...[truncated 4929659 chars]...
gService.java:519 - Waiting for messaging service to quiesce

INFO  [node1_isolatedExecutor:10] 2022-03-19 16:21:55,223 SubstituteLogger.java:169 - INFO  [node1_isolatedExecutor:10] node1 2022-03-19 16:21:55,221 MessagingService.java:519 - Waiting for messaging service to quiesce

INFO  [node2_isolatedExecutor:8] 2022-03-19 16:21:55,223 SubstituteLogger.java:169 - INFO  [node2_isolatedExecutor:8] node2 2022-03-19 16:21:55,222 MessagingService.java:519 - Waiting for messaging service to quiesce

Failures can also be repeatedly hit with CircleCI test multiplexer:

https://app.circleci.com/pipelines/github/adelapena/cassandra/1394/workflows/8d40d44b-7ccb-40fe-82d5-37db0bb228a3.

The same test looks ok in 4.0, as suggested by Butler and this repeated Circle run.

Issue Links

• is caused by: CASSANDRA-17164 CEP-14: Paxos Improvements
• relates to: CASSANDRA-17462 Test Failure: org.apache.cassandra.distributed.test.CASTest.testSuccessfulWriteBeforeRangeMovement
• links to: CASSANDRA-17641 PR trunk