[CASSANDRA-19520] Test failure: upgrade_tests.py::upgrade_through_versions_test::test_bootstrap_multidc failing counter checks

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 5.1-alpha
• Component/s: Consistency/Bootstrap and Decommission, Feature/Counters, Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Challenging
• Discovered By: DTest
• Platform: All
• Since Version: 5.1
• Assignee: Unassigned
• Reporter: Caleb Rackliffe
• Created: 03/Apr/24 20:50
• Updated: 16/Apr/24 20:01

Description

This appears to be a new failure in test_bootstrap_multidc, around upgrade_scenario verifying counter updates. I can't reproduce locally, so perhaps there is a counter race condition lurking somewhere...

assert 249 == 248 +249 -248

self = <upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_4_0_x_To_indev_trunk object at 0x7fbae9b96550>

def test_bootstrap_multidc(self):
# try and add a new node
# multi dc, 2 nodes in each dc
self.prepare()
cluster = self.cluster

if cluster.version() >= '5.0':
cluster.set_configuration_options({'user_defined_functions_threads_enabled': 'true',
'scripted_user_defined_functions_enabled': 'false'})
elif cluster.version() >= '3.0':
cluster.set_configuration_options({'enable_user_defined_functions': 'true',
'enable_scripted_user_defined_functions': 'true'})
elif cluster.version() >= '2.2':
cluster.set_configuration_options({'enable_user_defined_functions': 'true'})

cluster.populate([2, 2])
[node.start(use_jna=True, wait_for_binary_proto=True) for node in self.cluster.nodelist()]
self._multidc_schema_create()
> self.upgrade_scenario(populate=False, create_schema=False, after_upgrade_call=(self._bootstrap_new_node_multidc,))

upgrade_tests/upgrade_through_versions_test.py:865: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
upgrade_tests/upgrade_through_versions_test.py:506: in upgrade_scenario
call()
upgrade_tests/upgrade_through_versions_test.py:841: in _bootstrap_new_node_multidc
self._check_counters()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_4_0_x_To_indev_trunk object at 0x7fbae9b96550>

def _check_counters(self):
logger.debug("Checking counter values...")
session = self.patient_cql_connection(self.node2, protocol_version=self.protocol_version)
session.execute("use upgrade;")

for key1 in list(self.expected_counts.keys()):
for key2 in list(self.expected_counts[key1].keys()):
expected_value = self.expected_counts[key1][key2]

query = SimpleStatement("SELECT c from countertable where k1='{key1}' and k2={key2};".format(key1=key1, key2=key2),
consistency_level=ConsistencyLevel.ONE)
results = session.execute(query)

if results is not None:
actual_value = results[0][0]
else:
# counter wasn't found
actual_value = None

> assert actual_value == expected_value
E assert 249 == 248
E +249
E -248

upgrade_tests/upgrade_through_versions_test.py:789: AssertionError