[CASSANDRA-19882] Test Failure: bootstrap_test.TestBootstrap

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: CI
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned Assign to me
• Reporter: Ekaterina Dimitrova
• Created: 30/Aug/24 15:09
• Updated: 30/Aug/24 15:14

Description

Flaky tests in TestBootstrap test class:
1) test_consistent_range_movement_false_with_replica_down_should_succeed
Seen here: https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2731/workflows/b987651a-5280-4dd6-9895-3298456406c8/jobs/61573/tests#failed-test-0
onsistent_range_movement_false_with_replica_down_should_succeed

ccmlib.node.TimeoutError: 29 Aug 2024 23:39:02 [node3] after 90.11/90 seconds Missing: ['Starting listening for CQL clients'] not found in system.log:
 Head: INFO  [main] 2024-08-29 23:37:23,347 YamlConfigura
 Tail: ...ava:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
self = <bootstrap_test.TestBootstrap object at 0x7f36f0dee220>

    def test_consistent_range_movement_false_with_replica_down_should_succeed(self):
>       self._bootstrap_test_with_replica_down(False)

bootstrap_test.py:306: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
bootstrap_test.py:416: in _bootstrap_test_with_replica_down
    node3.start(wait_for_binary_proto=successful_bootstrap_expected,
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:928: in start
    self.wait_for_binary_interface(from_mark=self.mark)
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:711: in wait_for_binary_interface
    self.watch_log_for("Starting listening for CQL clients", **kwargs)
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:610: in watch_log_for
    TimeoutError.raise_if_passed(start=start, timeout=timeout, node=self.name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1724974652.1128614, timeout = 90
msg = "Missing: ['Starting listening for CQL clients'] not found in system.log:\n Head: INFO  [main] 2024-08-29 23:37:23,347...t.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
node = 'node3'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 29 Aug 2024 23:39:02 [node3] after 90.11/90 seconds Missing: ['Starting listening for CQL clients'] not found in system.log:
E            Head: INFO  [main] 2024-08-29 23:37:23,347 YamlConfigura
E            Tail: ...ava:74)
E           	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
E           	at java.base/java.lang.Thread.run(Thread.java:833)

../env3.8/lib/python3.8/site-packages/ccmlib/node.py:56: TimeoutError

2) test_shutdown_wiped_node_cannot_join

ccmlib.node.TimeoutError: 29 Aug 2024 23:56:24 [node4] after 600.58/600 seconds Missing: ['A node with address /127.0.0.4:7000 already exists, cancelling join'] not found in system.log:
 Head: INFO  [main] 2024-08-29 23:46:24,274 YamlConfigura
 Tail: ...age:1] 2024-08-29 23:56:22,505 NoSpamLogger.java:107 - This node is not in the CMS, can't generate a consistent log fetch response to /127.0.0.4:7000
self = <bootstrap_test.TestBootstrap object at 0x7f5c3fb40df0>

    def test_shutdown_wiped_node_cannot_join(self):
>       self._wiped_node_cannot_join_test(gently=True)

bootstrap_test.py:618: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
bootstrap_test.py:663: in _wiped_node_cannot_join_test
    node4.watch_log_for("A node with address {} already exists, cancelling join".format(node4.address_for_current_version_slashy()), from_mark=mark)
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:610: in watch_log_for
    TimeoutError.raise_if_passed(start=start, timeout=timeout, node=self.name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1724975183.919192, timeout = 600
msg = "Missing: ['A node with address /127.0.0.4:7000 already exists, cancelling join'] not found in system.log:\n Head: INF...SpamLogger.java:107 - This node is not in the CMS, can't generate a consistent log fetch response to /127.0.0.4:7000\n"
node = 'node4'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 29 Aug 2024 23:56:24 [node4] after 600.58/600 seconds Missing: ['A node with address /127.0.0.4:7000 already exists, cancelling join'] not found in system.log:
E            Head: INFO  [main] 2024-08-29 23:46:24,274 YamlConfigura
E            Tail: ...age:1] 2024-08-29 23:56:22,505 NoSpamLogger.java:107 - This node is not in the CMS, can't generate a consistent log fetch response to /127.0.0.4:7000

../env3.8/lib/python3.8/site-packages/ccmlib/node.py:56: TimeoutError

Issue Links

• Discovered while testing: CASSANDRA-18584 CEP-29: NOT operator