[CASSANDRA-17312] dtest-large.replace_address_test.TestReplaceAddress.test_restart_failed_replace (from Cassandra dtests)

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.27
• Component/s: Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: DTest
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra-dtest/commit/0d4fc63d7536616cf448a044d740955d308bc03a

Test and Documentation Plan

CI

• Assignee: Brandon Williams
• Reporter: Josh McKenzie

Authors

Brandon Williams

Reviewers

Berenguer Blasi

• Created: 27/Jan/22 17:55
• Updated: 14/Feb/22 15:18
• Resolved: 14/Feb/22 15:18

Description

Consistently failing on 3.0.x

https://ci-cassandra.apache.org/job/Cassandra-3.0/240/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace_2/

Failed 8 times in the last 16 runs. Flakiness: 73%, Stability: 50%

Error Message
ccmlib.node.TimeoutError: 26 Jan 2022 23:07:02 [replacement] after 90.12/90 seconds Missing: ['Starting listening for CQL clients'] not found in system.log: Head: INFO [main] 2022-01-26 23:04:33,906 YamlConfigura Tail: ...endingRangeCalculator:1] 2022-01-26 23:06:41,472 TokenMetadata.java:226 - Token -3193255413308472407 changing ownership from /127.0.0.3 to /127.0.0.4

Stacktrace
self = <replace_address_test.TestReplaceAddress object at 0x7f99546197c0>

    @since('2.2')
    @pytest.mark.resource_intensive
    def test_restart_failed_replace(self):
        """
            Test that if a node fails to replace, it can join the cluster even if the data is wiped.
            """
>       self._test_restart_failed_replace(mode='wipe')

replace_address_test.py:479: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
replace_address_test.py:539: in _test_restart_failed_replace
    self.replacement_node.start(jvm_args=["-Dcassandra.replace_address_first_boot={}"
../venv/lib/python3.8/site-packages/ccmlib/node.py:901: in start
    self.wait_for_binary_interface(from_mark=self.mark)
../venv/lib/python3.8/site-packages/ccmlib/node.py:689: in wait_for_binary_interface
    self.watch_log_for("Starting listening for CQL clients", **kwargs)
../venv/lib/python3.8/site-packages/ccmlib/node.py:588: in watch_log_for
    TimeoutError.raise_if_passed(start=start, timeout=timeout, node=self.name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1643238332.8472316, timeout = 90
msg = "Missing: ['Starting listening for CQL clients'] not found in system.log:\n Head: INFO  [main] 2022-01-26 23:04:33,906...26 23:06:41,472 TokenMetadata.java:226 - Token -3193255413308472407 changing ownership from /127.0.0.3 to /127.0.0.4\n"
node = 'replacement'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 26 Jan 2022 23:07:02 [replacement] after 90.12/90 seconds Missing: ['Starting listening for CQL clients'] not found in system.log:
E            Head: INFO  [main] 2022-01-26 23:04:33,906 YamlConfigura
E            Tail: ...endingRangeCalculator:1] 2022-01-26 23:06:41,472 TokenMetadata.java:226 - Token -3193255413308472407 changing ownership from /127.0.0.3 to /127.0.0.4

../venv/lib/python3.8/site-packages/ccmlib/node.py:56: TimeoutError

This test can be run isolation via 'pytest --force-resource-intensive-tests --cassandra-dir=~/cassandra replace_address_test.py::TestReplaceAddress::test_restart_failed_replace'