[CASSANDRA-19344] Range movements involving transient replicas must safely enact changes to read and write replica sets

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.1
• Component/s: CI
• Epic Link: CEP-21: Follow Up
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/dabcb175527d3c2daef54c6ce029b3c3054b2a77

Test and Documentation Plan

New and existing tests in attached CI results.
The few failures/errors in the attached results are known (CASSANDRA-19343) or look to be infra-related.

• Assignee: Sam Tunnicliffe
• Reporter: Ekaterina Dimitrova

Authors

Marcus Eriksson, Sam Tunnicliffe

Reviewers

Alex Petrov, Marcus Eriksson

• Created: 30/Jan/24 21:20
• Updated: 20/May/24 11:33
• Resolved: 19/Apr/24 08:42

Description

(edit) This was originally opened due to a flaky test org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode-_jdk17

The test can fail in two different ways:

junit.framework.AssertionFailedError: NOT IN CURRENT: 31 -- [(00,20), (31,50)] at org.apache.cassandra.distributed.test.TransientRangeMovementTest.assertAllContained(TransientRangeMovementTest.java:203) at org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode(TransientRangeMovementTest.java:183) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

as in here - https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2639/workflows/32b92ce7-5e9d-4efb-8362-d200d2414597/jobs/55139/tests#failed-test-0
and

junit.framework.AssertionFailedError: nodetool command [removenode, 6d194555-f6eb-41d0-c000-000000000003, --force] was not successful stdout: stderr: error: Node /127.0.0.4:7012 is alive and owns this ID. Use decommission command to remove it from the ring -- StackTrace -- java.lang.UnsupportedOperationException: Node /127.0.0.4:7012 is alive and owns this ID. Use decommission command to remove it from the ring at org.apache.cassandra.tcm.sequences.SingleNodeSequences.removeNode(SingleNodeSequences.java:110) at org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3682) at org.apache.cassandra.tools.NodeProbe.removeNode(NodeProbe.java:1020) at org.apache.cassandra.tools.nodetool.RemoveNode.execute(RemoveNode.java:51) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:388) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:373) at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:272) at org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:1129) at org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$51(Instance.java:1038) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) Notifications: Error: java.lang.UnsupportedOperationException: Node /127.0.0.4:7012 is alive and owns this ID. Use decommission command to remove it from the ring at org.apache.cassandra.tcm.sequences.SingleNodeSequences.removeNode(SingleNodeSequences.java:110) at org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3682) at org.apache.cassandra.tools.NodeProbe.removeNode(NodeProbe.java:1020) at org.apache.cassandra.tools.nodetool.RemoveNode.execute(RemoveNode.java:51) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:388) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:373) at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:272) at org.apache.cassandra.distributed.impl.Instance$DTestNodeTool.execute(Instance.java:1129) at org.apache.cassandra.distributed.impl.Instance.lambda$nodetoolResult$51(Instance.java:1038) at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:833) at org.apache.cassandra.distributed.api.NodeToolResult$Asserts.fail(NodeToolResult.java:214) at org.apache.cassandra.distributed.api.NodeToolResult$Asserts.success(NodeToolResult.java:97) at org.apache.cassandra.distributed.test.TransientRangeMovementTest.testRemoveNode(TransientRangeMovementTest.java:173) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

as in here - https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2634/workflows/24617d26-e297-4857-bc43-b6a04e64a6ea/jobs/54534/tests#failed-test-0

Issue Links

• Discovered while testing: CASSANDRA-19167 Test failure: CQLVectorTest fails with heap_buffers
• links to: GitHub Pull Request #3196