[CASSANDRA-17140] Broken test_rolling_upgrade - upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_3_0_x_To_indev_4_0_x

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.27, 3.11.13, 4.0.4, 4.1-alpha1, 4.1
• Component/s: CI
• Bug Category: Correctness - Test Failure
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra-dtest/commit/bcd094b19f8c771ea8fa4aad73fbfda72621eb4c

Test and Documentation Plan

See PR

• Assignee: Berenguer Blasi
• Reporter: Yifan Cai

Authors

Berenguer Blasi

Reviewers

Josh McKenzie

• Created: 12/Nov/21 00:04
• Updated: 14/Sep/23 05:37
• Resolved: 11/Apr/22 05:09

Description

The tests "test_rolling_upgrade" fail with the below error. 
 
https://app.circleci.com/pipelines/github/yifan-c/cassandra/279/workflows/6340cd42-0b27-42c2-8418-9f8b56c57bea/jobs/1990
 
I am able to alway produce it by running the test locally too. 
$ pytest --execute-upgrade-tests-only --upgrade-target-version-only --upgrade-version-selection all --cassandra-version=4.0 upgrade_tests/upgrade_through_versions_test.py::TestUpgrade_indev_3_11_x_To_indev_4_0_x::test_rolling_upgrade
 

self = <upgrade_tests.upgrade_through_versions_test.TestUpgrade_indev_3_0_x_To_indev_4_0_x object at 0x7ffba4242fd0>
subprocs = [<Process(Process-1, stopped[SIGKILL] daemon)>, <Process(Process-2, stopped[1] daemon)>]

    def _check_on_subprocs(self, subprocs):
        """
            Check on given subprocesses.
    
            If any are not alive, we'll go ahead and terminate any remaining alive subprocesses since this test is going to fail.
            """
        subproc_statuses = [s.is_alive() for s in subprocs]
        if not all(subproc_statuses):
            message = "A subprocess has terminated early. Subprocess statuses: "
            for s in subprocs:
                message += "{name} (is_alive: {aliveness}), ".format(name=s.name, aliveness=s.is_alive())
            message += "attempting to terminate remaining subprocesses now."
            self._terminate_subprocs()
>           raise RuntimeError(message)
E           RuntimeError: A subprocess has terminated early. Subprocess statuses: Process-1 (is_alive: True), Process-2 (is_alive: False), attempting to terminate remaining subprocesses now.

Issue Links

• Dependency: CASSANDRA-16262 4.0 Quality: Coordination & Replication Fuzz Testing
• is a parent of: CASSANDRA-17806 Flaky test_rolling_upgrade
• is duplicated by: CASSANDRA-17290 Test Failure: python dtest test_parallel_upgrade
• relates to: CASSANDRA-17086 Fix python dtest upgrade tests
• relates to: CASSANDRA-17296 Test Failure: dtest-upgrade.upgrade_tests.upgrade_through_versions_test.TestProtoV4Upgrade_AllVersions_RandomPartitioner_EndsAt_Trunk_HEAD.test_rolling_upgrade
• links to: CASSANDRA-17140 PR dtests