[CASSANDRA-19539] Test Failure: test_bootstrap_with_reset_bootstrap_state

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.x
• Component/s: CI, Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned
• Reporter: Ekaterina Dimitrova
• Created: 08/Apr/24 16:37
• Updated: 08/Apr/24 16:39

Description

Failing on trunk:

ccmlib.node.TimeoutError: 03 Apr 2024 19:41:13 [node3] after 180.22/180 seconds Missing: ['Starting listening for CQL clients'] not found in system.log:
 Head: INFO  [main] 2024-04-03 19:37:59,845 YamlConfigura
 Tail: ...19 - Got error from /127.0.0.1:7000: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.1:7000], checkLive=true}
self = <bootstrap_test.TestBootstrap object at 0x7f682b09ae20>

    @since('2.2')
    def test_bootstrap_with_reset_bootstrap_state(self):
        """Test bootstrap with resetting bootstrap progress"""
        cluster = self.cluster
        cluster.set_environment_variable('CASSANDRA_TOKEN_PREGENERATION_DISABLED', 'True')
        cluster.set_configuration_options(values={'stream_throughput_outbound_megabits_per_sec': 1})
        cluster.populate(2).start()
    
        node1 = cluster.nodes['node1']
        node1.stress(['write', 'n=100K', '-schema', 'replication(factor=2)'])
        node1.flush()
    
        # kill node1 in the middle of streaming to let it fail
        t = InterruptBootstrap(node1)
        t.start()
    
        # start bootstrapping node3 and wait for streaming
        node3 = new_node(cluster)
        try:
            node3.start()
        except NodeError:
            pass  # node doesn't start as expected
        t.join()
        node1.start()
    
        # restart node3 bootstrap with resetting bootstrap progress
        node3.stop(signal_event=signal.SIGKILL)
        mark = node3.mark_log()
        node3.start(jvm_args=["-Dcassandra.reset_bootstrap_progress=true"])
        # check if we reset bootstrap state
        node3.watch_log_for("Resetting bootstrap progress to start fresh", from_mark=mark)
        # wait for node3 ready to query, 180s as the node needs to bootstrap
>       node3.wait_for_binary_interface(from_mark=mark, timeout=180)

bootstrap_test.py:513: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:709: in wait_for_binary_interface
    self.watch_log_for("Starting listening for CQL clients", **kwargs)
../env3.8/lib/python3.8/site-packages/ccmlib/node.py:608: in watch_log_for
    TimeoutError.raise_if_passed(start=start, timeout=timeout, node=self.name,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

start = 1712173092.936025, timeout = 180
msg = "Missing: ['Starting listening for CQL clients'] not found in system.log:\n Head: INFO  [main] 2024-04-03 19:37:59,845...00: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.1:7000], checkLive=true}\n"
node = 'node3'

    @staticmethod
    def raise_if_passed(start, timeout, msg, node=None):
        if start + timeout < time.time():
>           raise TimeoutError.create(start, timeout, msg, node)
E           ccmlib.node.TimeoutError: 03 Apr 2024 19:41:13 [node3] after 180.22/180 seconds Missing: ['Starting listening for CQL clients'] not found in system.log:
E            Head: INFO  [main] 2024-04-03 19:37:59,845 YamlConfigura
E            Tail: ...19 - Got error from /127.0.0.1:7000: TIMEOUT when sending TCM_COMMIT_REQ, retrying on CandidateIterator{candidates=[/127.0.0.1:7000], checkLive=true}

../env3.8/lib/python3.8/site-packages/ccmlib/node.py:56: TimeoutError

https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2680/workflows/8b1c0d0a-7458-4b43-9bba-ac96b9bfe64f/jobs/58929/tests#failed-test-0
https://ci-cassandra.apache.org/job/Cassandra-trunk/1859/#showFailuresLink

Issue Links

• Discovered while testing: CASSANDRA-19428 Clean up KeyRangeIterator classes