[CASSANDRA-17119] Broken test_speculative_data_request

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.2, 4.1-alpha1, 4.1
• Component/s: Test/dtest/python
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.1

Source Control Link

https://github.com/apache/cassandra-dtest/commit/01a647f4a1ec42d42198fe4c49fe654a7705d7d7

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Berenguer Blasi

Authors

Berenguer Blasi

Reviewers

Brandon Williams

• Created: 04/Nov/21 09:26
• Updated: 27/May/22 19:25
• Resolved: 15/Nov/21 08:46

Description

This has been failing consistently lately

Error Message

KeyError: '127.0.0.3'

Stacktrace

self = <read_repair_test.TestSpeculativeReadRepair object at 0x7f979a965850>

    @since('4.0')
    def test_speculative_data_request(self):
        """ If one node doesn't respond to a full data request, it should query the other """
        node1, node2, node3 = self.cluster.nodelist()
        assert isinstance(node1, Node)
        assert isinstance(node2, Node)
        assert isinstance(node3, Node)
        session = self.get_cql_connection(node1, timeout=2)
    
        session.execute(quorum("INSERT INTO ks.tbl (k, c, v) VALUES (1, 0, 1)"))
    
        node2.byteman_submit(['./byteman/read_repair/stop_writes.btm'])
        node3.byteman_submit(['./byteman/read_repair/stop_writes.btm'])
    
        session.execute("INSERT INTO ks.tbl (k, c, v) VALUES (1, 1, 2)")
    
        # re-enable writes
        node2.byteman_submit(['-u', './byteman/read_repair/stop_writes.btm'])
    
        node1.byteman_submit(['./byteman/read_repair/sorted_live_endpoints.btm'])
        node1.byteman_submit(['./byteman/request_verb_timing.btm'])
    
        with StorageProxy(node1) as storage_proxy:
            assert storage_proxy.blocking_read_repair == 0
            assert storage_proxy.speculated_rr_read == 0
            assert storage_proxy.speculated_rr_write == 0
    
            session = self.get_cql_connection(node1)
            node2.byteman_submit(['./byteman/read_repair/stop_data_reads.btm'])
            results = session.execute(quorum("SELECT * FROM ks.tbl WHERE k=1"))
    
            timing = request_verb_timing(node1)
>           repair_req_node3 = timing[node3.ip_addr].get('READ_REPAIR_REQ')
E           KeyError: '127.0.0.3'

read_repair_test.py:621: KeyError

Issue Links

• Discovered while testing: CASSANDRA-16928 InetAddressAndPort to extend InetAddress
• links to: CASSANDRA-17119 PR dtests