[CASSANDRA-17296] Test Failure: dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD.test_rolling_upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.30, 4.0.12, 4.1.4, 5.0-alpha2, 5.0, 5.1
• Component/s: Test/dtest/python
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.1

Source Control Link

https://github.com/apache/cassandra-dtest/commit/ec3866d9aec0288b6ac479078e298a05c5df6d23

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Josh McKenzie

Authors

Berenguer Blasi

Reviewers

Brandon Williams

• Created: 26/Jan/22 17:38
• Updated: 01/Nov/23 08:54
• Resolved: 20/Sep/23 05:08

Description

2 failures in 30, looks flaky on timing / subprocess termination.

https://ci-cassandra.apache.org/job/Cassandra-trunk/920/testReport/dtest-upgrade.upgrade_tests.upgrade_through_versions_test/TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD/test_rolling_upgrade/

Failed 2 times in the last 30 runs. Flakiness: 10%, Stability: 93%
Error Message
RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.
Stacktrace
self = <upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD object at 0x7f22685cebb0>

@pytest.mark.timeout(3000)
def test_rolling_upgrade(self):
"""
Test rolling upgrade of the cluster, so we have mixed versions part way through.
"""
> self.upgrade_scenario(rolling=True)

upgrade_tests/upgrade_through_versions_test.py:320:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
upgrade_tests/upgrade_through_versions_test.py:398: in upgrade_scenario
self._check_on_subprocs(self.fixture_dtest_setup.subprocs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD object at 0x7f22685cebb0>
subprocs = [<Process name='Process-1' pid=28667 parent=314 stopped exitcode=-SIGKILL daemon>, <Process name='Process-2' pid=28686 parent=314 stopped exitcode=1 daemon>]

def _check_on_subprocs(self, subprocs):
"""
Check on given subprocesses.

If any are not alive, we'll go ahead and terminate any remaining alive subprocesses since this test is going to fail.
"""
subproc_statuses = [s.is_alive() for s in subprocs]
if not all(subproc_statuses):
message = "A subprocess has terminated early. Subprocess statuses: "
for s in subprocs:
message += "

{name}

(is_alive:

{aliveness}

), ".format(name=s.name, aliveness=s.is_alive())
message += "attempting to terminate remaining subprocesses now."
self._terminate_subprocs()
> raise RuntimeError(message)
E RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

upgrade_tests/upgrade_through_versions_test.py:456: RuntimeError

Issue Links

• is related to: CASSANDRA-17140 Broken test_rolling_upgrade - upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_3_0_x_To_indev_4_0_x
• links to: CASSANDRA-17296 dtests