[CASSANDRA-16381] nodetool removenode error “Conflicting replica added”

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.0-rc1, 4.0
• Component/s: Consistency/Bootstrap and Decommission
• Bug Category: Correctness - API / Semantic Definition
• Severity: Critical
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/64f54f9fb0ac1fe2920f44379326cf076ab8aab8

Test and Documentation Plan

dtest offered by Brandon.

• Assignee: Michael Semb Wever
• Reporter: vincent royer

Authors

Michael Semb Wever

Reviewers

Adam Holmberg, Berenguer Blasi, Brandon Williams

• Created: 12/Jan/21 16:50
• Updated: 02/Jun/21 18:11
• Resolved: 03/Mar/21 07:16

Description

When testing elassandra on C* 4.0, integration tests with ccm systematically failed on removing a node with the following error “Conflicting replica added” . This integration test  was ok with Elassandra based on Cassandra 3.11, and there is no changes in that test. Moreover, it seems there is no cassandra-test (dtest) for removing a node (there is only one removenode test for transient replication). The topology_test.py remove a node from the CCM cluster, but it does not call nodetool removenode.

I wonder if we have a non-tested regression here in C 4.0 ?

++ ccm node1 nodetool status
++ awk ‘/127.0.0.3/ \{ print $7 }’
+ HOST_ID3=6d2e858f-dacc-4c7c-a626-14b45f6b3b94
+ ccm node3 stop
+ ccm node1 nodetool removenode 6d2e858f-dacc-4c7c-a626-14b45f6b3b94
Traceback (most recent call last):
  File “/usr/local/bin/ccm”, line 4, in <module>
    __import__(‘pkg_resources’).run_script(‘ccm==3.1.6’, ‘ccm’)
  File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py”, line 742, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File “/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py”, line 1674, in run_script
    exec(script_code, namespace, namespace)
  File “/Library/Python/2.7/site-packages/ccm-3.1.6-py2.7.egg/EGG-INFO/scripts/ccm”, line 112, in <module>  File “build/bdist.macosx-10.14-intel/egg/ccmlib/cmds/node_cmds.py”, line 233, in run
  File “build/bdist.macosx-10.14-intel/egg/ccmlib/node.py”, line 848, in nodetool
  File “build/bdist.macosx-10.14-intel/egg/ccmlib/node.py”, line 2131, in handle_external_tool_process
ccmlib.node.ToolError: Subprocess [‘nodetool’, ‘-h’, ‘localhost’, ‘-p’, ‘7100’, ‘removenode’, ‘6d2e858f-dacc-4c7c-a626-14b45f6b3b94’] exited with non-zero status; exit status: 1;
stdout: nodetool: Conflicting replica added (expected unique ranges): Full(/127.0.0.1:7000,(4949329179655327935,6135417578204142297]); existing: Full(/127.0.0.1:7000,(4949329179655327935,6135417578204142297])
See ‘nodetool help’ or ‘nodetool help <command>’.++ finish
++ echo ‘ERROR occurs, test failed’
ERROR occurs, test failed
++ ‘[’ ‘!’ -z ‘’ ‘]’
++ exit 1

Issue Links

• causes: CASSANDRA-16525 Gossip STATUS can be either missing during upgrade or stale after upgrade